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Abstract 


As  a  result  of  recommendations  made  by  the  National  Research  Council  to  Mr.  C,  Adolph, 
OSD  Director,  Test  and  Evaluation  (DT&E),  in  a  r^ort,  “Vulnerability  Assessment  of  Aircraft-A 
Review  of  the  DOD  LFT&E  Program,”  January  1993,  an  effort  was  launched  to  improve  the  live 
fire  test  (UFT)  waiver  process  through  development  of  a  methodology  to  quantify  advantages  and 
disadvantages  of  full-scale,  full-up  test  programs.  Mr.  Walter  Hollis,  the  Army  Deputy  Under 
Secretary  for  Operations  Research  (DUSA[OR]),  tasked  the  U.S.  Army  Materiel  Systems 
Analysis  Activity  (AMSAA)  and  U.S.  Army  Research  Laboratory  (ARL)  to  develop  the  Live 
Fire  Vulnerability/Lefliality  Risk-Benefit  Assessment  Methodology.  ARL  (Deitz  et  al.)  was 
chosen  to  lead  this  effort  and  AMSAA  (LaGrange  et  al.)  was  to  provide  support.  On  22  Mar  93, 
the  set  of  proposed  deliverables  was  accepted  by  Mr.  Adolph  and  the  Senior  Test  and  Evaluation 
Group.  This  report  presents  a  history  and  summarizes  work  done  by  the  group.  Section  2 
discusses  in  detail  the  V/L  process  and  mediodology  and  models  used  in  V/L  assessments. 
Section  3  examines  the  connection  of  damage-to-engineering  operations  to  processes  different 
fi-om  standard  V/L  analyses.  Section  4  discusses  the  type  of  survivability/vulnerability  information 
that  must  be  developed  during  various  phases  of  the  acquisition  cycle.  Section  5  discusses  four 
methodologies  that  could  be  applied  to  assess  the  cost-benefit  LFT.  Section  6  summarizes 
conclusions  and  recommendations.  Appendices  A-D  provide  details  on  the  supporting  theorems 
of  section  4. 
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1.  INTRODUCTION 


1.1  Background.  In  1991,  Mr.  Charles  Adolph,  Director,  Test  and  Evaluation  (T&E),  Office  of  the 
Secretary  of  Defense  (OSD),  tasked  the  National  Research  Council  (NRC)  to  review  the  Department  of 
Defense  (DOD)  Live-Fire  Test  and  Evaluation  (LFT&E)  Program.  Specifically,  NRC  was  asked  to  review 
several  methodologies,  such  as  computer  modeling,  engineering  analysis,  and  live-fire  testing  (LFT),  and 
identify  and  evaluate  the  costs  and  effectiveness  of  these  methodologies  and  identify  deficiencies. 
Recommendations  were  requested.  NRC  completed  their  review  in  late  1992,  and  their  findings  and 
recommendations  were  published  in  January  1993  in  a  report  entitled  "Vulnerability  Assessment  of 
Aircraft  -  A  Review  of  the  DOD  LFT&E  Program." 

LFT  legislation  requires  full-up  LFT  for  covered  systems,  unless  a  waiver  of  full-up  LFT  is  requested 
before  Milestone  II  (MS  II).  If  a  waiver  of  full-up  LFT  is  requested,  it  must  include  plans  for  assessing 
the  vulnerability/lethality  (V/L)  of  the  covered  system  and  the  waiver  must  show  why  full-up  LFT  would 
be  "unreasonably  expensive  and  impractical."  The  waiver  must  include  "a  report  explaining  how  the 
Secretary  plans  to  evaluate  the  survivability  or  the  lethality  of  the  system  or  program  and  assessing 
possible  alternatives  to  realistic  survivability  testing  of  (he  system  or  program."  The  LFT  law  states  that 
a  covered  system  should  be  subjected  to  threat  weapons  "likely  to  be  encountered  in  combat." 

The  NRC  report  questioned  the  adequacy  of  the  1988  LFT&E  guidelines,  which  were  published  by 
OSD.  NRC  stated  that  the  waiver  process  was  not  understood  by  the  field  and  that  OSD  was  not  strictly 
enforcing  the  full-up  LFT  waiver  process.  The  NRC  concluded  that  no  covered  system  had  ever 
undergone  fiill-up  LFT  and  that  no  system  had  ever  requested  a  waiver  from  full-up  LFT.  The  1988  OSD 
LFT&E  guidelines  do  not  stress  the  need  to  request  a  waiver  from  full-up  LFT  (when  less  than  full-up 
testing  is  planned)  but  imply  that  other  methods  of  testing  and  analysis  can  satisfy  the  LFT  legislative 
requirements  (without  having  to  formally  request  a  waiver).  The  NRC  report  concluded  that  the  field  did 
not  understand  the  LFT  waiver  process  and  that  systems  managers  were  reluctant  to  request  a  waiver  for 
fear  that  it  would  "stigmatize"  their  programs.  Also,  there  was  a  general  lack  of  understanding  as  to  what 
constituted  realistic  smvivability  testing  (especially  as  it  related  to  threat  weapons  expected  to  be 
encountered  in  combat). 

The  conclusions  section  of  the  NRC  report  states  that  "the  committee  believes  that  the  1987 
Congressional  LFT  law  mandates  LFT  of  full-scale,  full-up  aircraft,  including  onboard  ordnance,  unless 
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a  waiver  is  granted  by  the  Secretary  of  Defense."  The  committee  also  concluded  "that  a  waiver  is  required 
to  omit  the  full-scale,  full-up  tests." 

One  of  the  recommendations  made  by  the  NRC  was  that  "the  T&E  Director  formalize  the  LFT  waiver 
process  by  developing  a  risk-benefit  assessment  methodology  that  can  be  used  uniformly  to  determine 
whether  a  full-scale,  fiill-up  test  program  for  any  particular  aircraft  is  "uiureasonably  expensive  and 
impractical."  The  methodology  must  also  apply  to  the  evaluation  of  the  alternate  LFT  program  for  the 
subscale  targets."  Mr.  Charles  Adolph  accepted  this  NRC  recommendation  and  on  27  January  1993 
prepared  a  memorandum  to  the  Senior  T&E  Group  requesting  the  development  of  a  risk-benefit 
assessment  methodology  to  be  applied  uniformly  when  deciding  whether  full-scale,  system-level  testing 
is  "unreasonably  expensive  and  impractical,"  and  a  formalized  waiver  process  that  supports  this 
methodology. 

An  OSD  "Committee  on  Vulnerability  Guidelines  (CVG)"  was  formed  to  address  the  issues  raised  by 
Mr.  Adolph.  The  committee  is  chaired  by  Dr.  A1  Rainis  (Tactical  Systems/OSD[A])  and  has  membership 
from  each  of  the  services,  from  SDIO,  and  from  the  office  of  Mr.  O’Bryon,  Deputy  Director,  T&E,  Land 
&  Maritime  Systems  (DT&E/LMS),  and  Mr.  Ledesma,  Deputy  Director,  T&E,  Air  &  Space  Program 
(DT&E/ASP).  Dr.  Rainis  met  with  Mr.  Adolph’s  Senior  T&E  Group  in  early  February  1993  and  was 
tasked  to  prepare  a  Scope-of-Work  (SOW)  to  develop  the  "risk/benefit  assessment  methodology."  The 
CVG  prepared  the  Risk/Benefit  SOW  and  briefed  it  to  the  Senior  T&E  Group  on  24  February  1993.  The 
Army  member  on  the  Senior  T&E  Group,  Mr.  Walter  Hollis,  the  Deputy  Under  Secretary  for  Army 
Operations  Research  (DUSA[OR])  and  Army  member  of  Mr.  Charles  Adolph’s  Senior  T&E  Group,  felt 
that  no  contractor  had  the  necessary  LFT  experience  and  expertise  to  perform  this  effort  and  recommended 
that  the  task  be  assigned  to  the  Army.  The  Senior  T&E  Group  concurred  with  this  approach  and  on 
26  February  1993  Mr.  Hollis  tasked  the  U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA)  and 
the  U.S.  Army  Research  Laboratory  (ARL)  to  develop  the  "Live-Fire  (LF)  V/L  Risk-Benefit  Assessment 
Methodology."  A  draft  SOW  was  provided  that  contained  the  background,  technical  requirements,  and 
schedule. 

Following  discussions  between  Mr.  Myers  (Director,  AMSAA)  and  Mr.  Vitali  (Director,  ARL),  it  was 
agreed  that  ARL  (Dr.  Deitz  et  al.)  would  take  die  lead  on  this  effort  and  AMSAA  (Mr.  LaGrange  et  al.) 
would  provide  support. 
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The  risk-benefit  assessment  methodology  SOW  contained  the  following  tasks: 

(1)  The  contracted  agency  shall  conduct  a  technical  analysis  and  develop  a  risk-benefit  assessment 
methodology,  initially  to  address  each  of  the  three  basic  categories  of  weapon  systems,  namely, 
aircraft,  ships,  and  land  platforms,  which  quantifies  the  benefits  associated  with  fiill-up, 
system-level  live-fire  V/L  tests,  the  risks  associated  with  waiving  these  tests,  plus  the  risk-benefit 
of  implementing  each  of  the  following  alternatives: 

•  No  testing  or  minor  testing  of  components  only. 

•  Live-fire  V/L  testing  of  major  subassemblies  and  assemblies  only. 

•  Live-fire  V/L  testing  of  the  total  system  only. 

•  Live-fire  V/L  testing  of  the  total  system  with  testing  of  major  subassemblies  and  assemblies. 

(2)  The  technical  analysis  shall  consider,  time  permitting,  the  effectiveness  of  existing  vulnerability 
analysis  methods  used  by  each  of  the  Services’  development  agencies  and  how  these  methods 
should  contribute  to  the  risk-benefit-cost  balance  assessment  methodology. 

(3)  The  methodology  shall  consider  the  likelihood  of  an  undetected  vulnerability  and  compare  the 
risk-cost-payoff  of  the  different  alternatives,  described  previously,  with  the  likelihood  of  detecting 
a  vulnerability  for  which  a  cost-effective  fix  would  be  practical.  In  developing  the  methodology, 
the  contracted  agency  shall  include  considerations  of  the  susceptibility  (probability  of  hit)  of  the 
system,  and  test  facility  capabilities. 

(4)  The  methodology  shall  also  address  the  risk-cost-payoff  to  the  system  acquisition  strategy  related 
to  the  alternatives,  listed  previously,  when  fixes/design  changes  are  required.  Further,  the 
contracted  agency  shall  include  a  determination  of  the  usefulness  and  applicability  of  transfer  of 
data  from  other  relevant  test  programs. 

(5)  The  contracted  agency  shall  provide  a  "best  effort"  to  include  in  the  methodology  the  benefits  of 
those  unquantifiables  not  specifically  listed  in  the  previous  approaches. 
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(6)  Based  on  the  previous  analysis,  the  contracted  agency  shall  develop  risk-benefit  assessment 
methodologies  applicable  to  the  three  basic  categories  of  weapon  systems,  and,  if  possible, 
combine  them  into  a  single  risk-benefit  assessment  methodology  that  can  be  applied  to  all  classes 
of  weapon  systems. 

1.2  Approach.  On  1  March  1993,  ARL  responded  to  the  Mr.  Walter  Hollis  task  and  agreed  to  take 
the  lead  in  developing  a  "Live  Fire  V/L  Risk-Benefit  Assessment  Methodology"  and  AMSAA  agreed  to 
provide  support.  On  4  March  1993,  Dr.  Deitz  (ARL)  and  Mr.  LaGrange  (AMSAA)  met  separately  with 
Dr.  A1  Rainis  and  Mr.  Walter  Hollis.  Dr.  Rainis  and  Mr.  Hollis  provided  guidance  regarding  the  scope 
and  schedule  for  developing  the  LFT  V/L  Risk-Benefit  Methodology.  It  was  agreed  that  a  final  draft 
report  would  be  prepared  by  the  end  of  July  1993  and  that  progress  briefings  would  be  provided  to 
Mr.  Adolph’s  Senior  T&E  Group  when  requested. 

On  9  March  1993,  Mr.  LaGrange  and  Dr.  Deitz  provided  a  background  briefing  to  AMSAA 
management  and  outlined  the  approach  to  be  taken  on  this  study.  On  16  March  1993,  Dr.  Deitz  and 
Mr.  LaGrange  briefed  the  planned  approach  to  the  tri-service  CVG  group  headed  by  Dr.  Rainis.  On 
19  March  1993,  ARL  and  AMSAA  management  was  given  a  pre-brief  of  the  planned  study  approach. 

On  22  March  1993,  the  ARL/AMSAA  "Live-Fire  V/L  Risk-Benefit  Methodology"  was  briefed  to 
Mr.  Charles  Adolph  and  the  Senior  T&E  Group.  This  briefing  reviewed  the  statement  of  work,  discussed 
a  top-down  and  bottom-up  approach  to  V/L  assessment,  and  proposed  a  set  of  deliverables  that  would 
result  from  this  study.  The  set  of  deliverables  focuses  on  the  following  four  areas: 

(1)  Describe  the  V/L  process  (V/L  taxonomy),  particularly  Levels  1  through  3,  discussing  how  testing 
and  modeling  measures  must  be  comparable  to  have  utility  and  how  the  mapping  operations  affect 
the  fidelity  of  the  predicted  measures. 

(2)  Document  the  building-block  process  so  as  to  reflect  the  full  gamut  of  tests/models/decision 
processes  to  provide  an  acceptable  basis  for  the  LF  assessment  process.  For  each  element/task 
of  the  process,  provide  a  description  of  required  standards  and  practices  to  ensure  adequate 
fidelity.  Examine  the  applicability  of  the  process  for  a  range  of  target  classes  and  weapons 
programs. 
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(3)  Investigate  the  nature  of  various  specific  cost/risk/benefit  meastires.  Identity  which  might  have 
promising  utility.  Develop  various  methods  (either  by  sampling  statistics  or  heuristic  approaches) 
to  relate  data/process  uncertainties  to  downstream  measures.  This  effort  should  include  the 
development  of  a  specific  decision  process,  applicable  at  Acquisition  MS  11,  which  will 
aid/assist/illuminate  the  risk/benefit  factors  involved  in  the  decision  whether  or  not  to  request  a 
final  (or  conditional)  waiver  for  combat-configured  LFT. 

(4)  Use  the  V/L  process  (of  Effort  1)  to  show  the  connectivity  of  Damage- to-Engineering  Operations 
among:  (a)  the  LF  assessment  process,  (b)  the  battle  damage  assessment  and  repair  (BDAR) 
process,  (c)  the  reliability,  availability,  and  maintainability  (RAM)  process,  and  (d)  the  operational 
and  developmental  test  processes. 

This  generalized  approach  was  accepted  by  Mr.  Adolph’s  Senior  T&E  group. 

The  next  five  sections  summarize  the  work  performed  in  each  of  the  four  areas.  Section  2  presents 
in  detail  the  V/L  taxonomy  and  the  methodology  and  models  used  to  do  V/L  assessments.  Section  3 
examines  the  connectivity  of  damage-to-engineering  operations  with  processes  different  from  standard  V/L 
analyses.  Section  4  discusses  the  type  of  survivability  and  vulnerability  information  fliat  must  be 
developed  during  various  phases  of  the  acquisition  cycle.  Section  5  discusses  four  methodologies  that 
could  be  applied  to  assess  the  cost  benefits  of  doing  full-up  LFT  (and  the  risks  of  not  doing  full-up  LFT). 
Section  6  summarizes  conclusions  and  recommendations.  In  addition.  Appendices  A-C  provide  details 
on  the  supporting  theorems  of  section  4. 

2.  V/L  PROCESS 

To  understand  the  nature  of  V/L  assessment  of  targets,  it  is  critical  to  understand  the  framework  within 
which  all  assessments  for  the  past  40  years  have  taken  place.  This  framework  is  not  just  implicit  to 
computer-based  assessments  but  provides  a  key  link  to  field  observables  as  well.  The  vulnerability  process 
can  be  thought  of  as  a  transformation  of  information  among  four  levels. 
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2.1  V/L  Spaces.  Mappings,  and  Modeling. 


2.1.1  Basic  Definitions*  of  Lethality,  Vulnerability,  and  Survivability. 

Vulnerability:  The  characteristics  of  a  system  that  cause  it  to  suffer  a  degradation  (loss  or  reduction 
of  capability  to  perform  the  designated  mission[s])  as  a  result  of  having  been  subjected  to  a  hostile 
enviromnent  on  the  battlefield. 

Lethality:  The  ability  of  a  system  to  cause  the  loss  of,  or  a  degradation  in,  the  ability  of  a  target 
system  to  complete  its  designated  mission(s). 

Survivability:  The  capability  of  a  system  (resulting  from  the  synergism  among  personnel,  materiel, 
design,  tactics,  and  doctrine)  to  avoid,  withstand,  or  recover  in  hostile  (man-made  and  natural) 
environments  without  suffering  an  abortive  impairment  of  its  ability  to  accomplish  its  designated  mission. 
If  the  two  facets  under  control  of  a  weapons  designer,  materiel  and  design,  are  lumped  into  System 
Characteristics,  and  die  effects  of  personnel  are  distributed  appropriately  over  the  three  remaining  variables 
of  Characteristics,  Tactics,  and  Doctrine,  then  survivability  can  be  written  functionally  as: 

SURV  =  f  {  Threat  (Char,  Tactics,  Doc), 

Batdefield  Environment, 

System  (Char,  Tactics,  Doc)  }. 

2.1.2  V/L  Taxonomy.  Insight  into  these  definitions  can  be  gained  through  use  of  what  is  now  called 
the  V/L  Taxonomy.  It  was  first  generated  (Deitz  and  Ozolins  1988)  as  a  by-product  of  a  program  to 
improve  the  quality  of  LF  vulnerability  modeling.  In  essence,  the  taxonomy  provides  a  method  to 
decompose  the  elements  of  V/L  into  a  sequence  of  simpler  constituent  parts.  As  we  will  see,  the  parts 
relate  to  each  other  in  a  specific  processing  order,  but  are  fundamentally  different,  one  from  the  other,  and 
each  has  its  imique  and  appropriate  use  in  the  general  scheme  of  V/L  assessment. 

2. 1.2.1  V/L  Taxonomy  via  a  Combat  Analogue.  The  taxonomy  can  probably  best  be  introduced  via 
a  description  in  terms  of  its  physical  and  engineering  processes.  Figure  1  illustrates  such  a  view  of  the 
taxonomy.  TTie  so-called  process  structure  is  illustrated  with  a  missile  attacking  an  aircraft,  although 


*  Consistent  with  DODI  5000.2,  Defense  Acquisition  Management  Policies  and  Procedures.  23  February  1991. 
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Figure  1.  The  V/L  taxonomy  illustrated  with  physical  and  engineering  processes  in  the  center  column. 

Levels  0-4  are  described  on  the  left.  The  transformation  processes  between  the  levels  are 
described  on  the  right. 
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this  process  is  applicable  for  any  ballistic  threat  against  any  target.*  The  cartoons  at  the  center  of 
Figure  1  alternately  represent  the  levels,  indicated  on  the  left,  and  the  transformation  processes,  indicated 
on  the  right. 

We  start  with  an  explanation  of  the  process  of  vulnerability  using  Figure  1.  In  conventional 
vulnerability,  it  is  normal  practice  to  assume  a  warhead  hit  (or  ballistic  interaction)  with  a  target  as  a 
given.  It  is  useful  to  think  in  this  context  of  an  LF  test.  Level  1  represents  a  complete  geometric  and 
material  description  of  a  threat  (here  a  missile),  a  target  (here  an  aircraft),  and  the  relevant  kinematics  as 
the  two  just  begin  to  interact.  The  event  may  be  signaled  by  the  instant  an  unguided  bullet  begins  to 
collide  with  a  target  or,  as  implied  by  the  illustration,  a  fuze  triggers  an  explosive  mechanism  in  the 
vicinity  of  the  target.  The  process  of  an  LF  test  is  to  transform  an  undamaged  target  at  Level  1  to  a 
damaged  target  at  Level  2.  The  transformation  process  is  the  LF  event  itself  and  is  characterized  by  all 
the  physical  mechanisms  of  destruction.  They  may  include  main  penetrators,  fragments,  blast,  shock,  fitre, 
fumes,  and  even  synergistic  effects.  As  a  consequence  of  the  LF  transformation  event,  target  damage  may 
have  occurred  at  Level  2.  We  choose  to  think  of  Level  2  as  characterized  by  a  list  of  killed  components; 
sometimes  this  is  called  a  damage  vector. 

A  target  that  has  received  damage  may  likely  not  continue  to  operate  as  before  damage.  In  the  case 
of  a  damaged  aircraft  illustrated  here,  parts  of  the  control  surfaces  may  be  removed,  hydraulic  lines 
severed,  and  electronic  boxes  impaired.  In  a  test  which  might  be  performed,  the  rate  of  climb  might  be 
measured  to  see  how  this  key  capability  property  may  have  been  reduced.  The  capability  test  is  the 
transformation  process,  which  takes  target  damage  at  Level  2  and  transforms  it  to  reduced  capability  at 
Level  3.  The  transformation  from  Level  2  to  Level  3  can  be  thought  of  as  characterized  by  engineering 
relations.  It  is  important  to  note  that  the  metrics  of  Level  1,  threat^target  initial  conditions.  Level  2, 
damaged  components,  and  Level  3,  measures  of  capability,  are  all  measurable  and  objective.  A 
methodology  that  predicts  tire  inflation  levels  as  a  function  of  time  and  damage  that  can  then  be  combined 
with  scenario  requirements  to  determine  effectiveness  is  given  in  Grote,  Moss,  and  Davisson  (1996). 

The  capability  state  of  Level  3  should  be  characterized  by  all  capability  measures  which  cause  a 
military  platform  to  have  military  utility  or  worth  in  a  particular  mission.  For  example,  if  the  platform 
can  move,  the  metrics  might  include  measures  of  speed  and  agility.  If  the  platform  has  a  gun,  the  metrics 
might  include  time  to  acquire  a  target,  rate  of  fire,  and  hit  dispersion. 


*  It  will  be  noted  later  that  this  applied  as  well  to  nonballistic  threats  such  as  directed  energy  and  chemical  weapons  analyses. 
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The  final  transformation  occius  as  a  platform  with  reduced  measures-of-capability  is  exercised  in  a 
particular  mission  scenario.  If  the  particular  reduced  measures-of-capability  are  unimportant  to  the  mission 
at  hand,  then  the  utility  of  the  platform  may  remain  high.  If  not,  the  utility  may  be  reduced,  even  drop 
to  zero.  The  notion  of  measures-of-effectives  or  utility  is  illustrated  as  a  Level  4  metric  and  would  be 
reached  through  an  operational  test  or  war  experience.  Given  the  complexity  of  this  transformation  and 
the  lack  of  real-world  repeatability,  we  claim  that  Level  4  metrics  are  essentially  not  observable,  but  rather 
must  be  inferred  through  war  games  or  developed  via  subjective  processes. 

One  of  the  key  insights  provided  by  taxonomy  is  that  the  discipline  of  vulnerability  ranges  over  three 
distinct  kinds  of  metrics,  damage,  capability,  and  utility,  and  great  care  must  be  exercised  to  see  that  these 
metrics  are  not  confused,  incorrectly  calculated,  or  improperly  applied. 

In  contrast  to  vulnerability,  in  which  a  threat  interaction  with  a  target  is  normally  assumed,  lethality 
includes  the  process  of  getting  the  threat  to  the  target.  Thus  Figure  1  includes  a  Level  0,  which  represents 
the  initial  conditions  for  the  launch  of  the  threat.  The  transformation  of  the  threat  at  Level  0  to  the  arrival 
at  the  target  (Level  1)  would  occur  here  as  the  firing  of  the  missile.  In  a  set  of  repeated  experiments,  a 
distribution  of  threat  arrival  conditions  could  be  generated.  One  particular  condition  at  a  time  might  be 
chosen  to  use  for  a  given  initial  threat-target  interaction  at  Level  1. 

2. 1.2.2  V/L  Taxonomy  via  a  Mapping  Abstraction.  The  taxonomy  is  useful  in  developing  the 
mathematical  abstractions  needed  for  V/L  modeling.  Each  of  the  levels  of  the  process  can  be  thought  of 
as  a  mathematical  space.  As  illustrated  in  Figure  2,  the  cartoons  in  the  middle  of  Figure  1  describing  the 
levels  have  been  replaced  with  ellipses  representing  these  spaces.  The  information  at  Levels  0-4  can  each 
be  described  by  vectors  within  these  spaces,  here  represented  as  bullets;  however,  the  properties  of  the 
vectors  are  completely  different  from  one  space  to  the  next.  As  mentioned  previously,  the  metrics  of 
damage,  performance,  and  utility  are  not  interchangeable. 

As  noted  previously,  an  LF  test  can  be  thought  of  as  a  mapping  from  Level  1  to  Level  2.  If  the  LF 
shot  were  repeated,  it  is  likely  that  random  physical  processes  could  lead  to  a  different  damage  vector, 
thus  a  different  vector  would  result.  LF  tests  and  modeling  efforts  have  shown  (Deitz  and  Ozolins  1988) 
that  the  outcome  for  many  LF  tests  can  exceed  10^  individual  damage  vectors.  The  high  dimensionality 
of  Level  2  space  is  at  the  core  of  the  difficulty  in  validating  V/L  models. 
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Figure  2. 


Ihe  V/L  taxonomy  illustrated  via  a  mapping  abstraction.  The  ellipses  in  the  middle  column 
represent  mathematical  spaces.  The  points  contained  within  represent  vectors.  The  arrows 
represent  mapping  operators,  which  take  a  vector  at  one  level  and  perform  a  mapping  to  the 
next  lower  level. 


The  transformation  processes  listed  at  the  right  of  Figure  1  have  been  reproduced  in  Figure  2  as  well. 
It  is  useful  to  think  of  the  transformations  as  mathematical  operators.  These  operators  operate  on 
information  on  one  level  to  yield  information  at  the  next.  A  nomenclature  that  has  been  adopted  is  to  use 
a  capital  O  (for  operator),  followed  by  subscripts  indicating  the  input  and  output  levels.  Thus  the  Oq  j 
Operator  represents  the  mapping  of  the  threat  from  launch  to  the  arrival  at  the  target.  The  Oj  2  Operator 
represents  the  damage  mapping  process  of  an  LF  test.  The  O2  3  Operator  represents  the  transformation 
to  reduced  capability  of  a  target  following  damage.  And  flie  O3  4  Operator  represents  the  transformation 
from  reduced  capability  to  military  utility  for  a  particular  mission  profile. 

2.1.2.3  Insights  Provided  by  the  V/L  Taxonomy.  There  are  a  number  of  important  aspects  of  the 
taxonomy,  particularly  for  understanding  the  definitions  for  vulnerability,  lethality,  and  survivability. 

(1)  The  V/L  metrics  associated  with  each  of  the  Levels  0-4  are  fundamentally  distinct.  That  is  to 
say,  component  damage  across  a  weapon  platform  is  different  from  potentially  diminished 
capability  of  the  platform  and  both  are  different  from  the  possible  reduction  in  military  utility  of 
the  platform.  Both  vulnerability  of  a  platform  or  the  lethality  of  a  weapon  can  be  defined  in  any 
combination  of  metrics  from  Levels  2,  3,  and/or  4. 

(2)  The  five  levels  are  sequential,  independent,  and  nonpermutable. 

(3)  Modem  V/L  modeling  schemes  must  track  the  taxonomy  so  that  Levels  2  and  3  metrics 
can  be  compared  with  the  results  of  field  tests. 

(4)  The  operators  are  generally  stochastic,  nonlinear,  and  noninvertible;  therefore,  multiple 
levels  (e.g..  Levels  2-4)  cannot  be  properly  mapped  in  a  single  process. 

(5)  In  certain  V/L  modeling  tasks,  the  mathematical  mapping  operators  must  use  stochastic 
processes  to  yield  accurate  results.  This  is  most  notably  tme  in  the  damage  operator, 

Oi,2,  where  simplifications  using  expected-value  transformations  have  been  shown  to 
yield  incorrect  results. 

(6)  Level  4  metrics,  essentially  batdefield  utilities,  cannot  be  observed  through  testing. 
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2.1. 2.4  Taxonomy  and  Survivability.  A  number  of  times  we  have  emphasized  the  importance  of  not 
confounding  the  different  kinds  of  V/L  metrics  associated  with  Levels  2,  3,  and  4.  The  data  associated 
with  a  Level  2  metric  is  straightforward — it  is  simply  an  accounting  of  damaged  or  killed  components. 
At  Level  3,  the  metrics  are  capability.  Capability  measures  can  be  defined  clearly  in  terms  of  measurables 
such  as  top  speed,  minimum  speed,  rate  of  acceleration,  rate  of  fire,  etc.  Level  4  metrics  may  best  be 
thought  of  as  utilities  and  therefore  dimensionless.  At  Level  4,  however,  the  metrics  become  whatever 
the  modeler  may  mean  them  to  be,  or  even  something  else. 

It  may  be  helpful  to  review  a  hypothetical  example  of  an  aircraft  performing  a  military  mission.  With 
a  view  to  Figure  1 ,  let  us  assume  that  a  missile  attack  has  led  to  severing  a  fuel  line  to  one  of  two  engines 
on  the  aircraft.  The  damage  vector  at  Level  2  is  damage  vector  of  one  element,  one  killed  fuel  line. 
Applying  the  capability  operator  O2  3  to  the  damage  vector  gives  the  following  result  at  Level  3;  the 
aircraft  is  able  to  fly  straight  and  level,  but  not  climb.  Now  we  examine  the  Military  Utility  Operator, 
O3  4.  To  apply  this  operator,  a  number  of  missions  must  be  defined.  In  one  mission,  it  might  be 
necessary  to  climb  rapidly  to  avoid  ground  ordnance.  In  mission  two,  it  might  only  be  necessary  to 
maintain  level  flight.  Thus  we  could  define  two  O3  4  mapping  operators.  In  the  case  of  the  first,  the 
damaged  aircraft  could  not  fiilfill  the  mission  and  would  have  a  utility  of  zero.  In  the  case  of  the  second 
mission,  the  mission  could  be  supported,  giving  a  utility  of  one.  One  can  envision  missions  which  when 
applied  against  partially  performing  platforms  would  result  in  partial  utility  (0.0  <  U  <  1.0).  Given  some 
set  of  mission  utilities,  it  is  then  possible  to  develop  an  expected  utility,  averaged  over  some  set  of 
missions. 

Often  utilities,  averaged  or  otherwise,  are  used  by  the  community  of  war  games.  The  utilities  are  often 
simply  defined  to  be  probabilities  of  a  certain  class  of  kill.  The  utility,  on  the  same  interval  as  a 
probability,  is  used  in  the  war  game  to  make  a  draw,  assuming  a  ballistic  encounter.  Based  on  the 
outcome,  the  platform  may  be  removed  from  the  conflict.  Potentially  three  errors  are  committed  by  this 
practice. 

(1)  Burning  a  vulnerability  metric  in  a  war  game  scenario  which  has  already  been  burned  by  a 
vulnerability  analyst:  The  war  gamer  should  take  the  capability  information  fixrm  Level  3  and 
play  that  characterization  of  the  platform  in  his  mission  encounter.  The  war  game  will  then 
define  the  utility  of  the  (damaged)  platform. 
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(2)  Averaging  two  or  more  utilities:  Often  averages  are  performed  over  the  outcomes  of 
multiple  binning  processes.  This  is  legitimate  mathematically.  However,  a  major 
problem  occurs  when  an  average  utility  is  applied  to  a  specific  mission.  They  may,  in 
fact,  be  very  different  numbers. 

(3)  Turning  a  utility  into  a  probability:  A  practice  which  has  seen  widespread  use  in  the 
groimd  arena  has  been  to  argue  that  an  average  battlefield  utility  is  equivalent  to  the 
probability  of  total  loss  of  the  modeled  capability. 

This  third  practice  has  been  shown  wanting  for  many  years  (Rapp  1983  and  Starks  1991),  but  the 
numbers  provided  by  the  V/L  community  to  its  customers  are  still  referred  to  as  "probabilities  of  kill"  or 
"expected  loss  of  function  (LOF),"  even  if  at  their  foundation,  they  may  suffer  from  some  or  all  of  these 
three  serious  problems. 

Finally,  if  a  series  of  utilities  are  derived  as  a  function  of  ballistic  threats  introduced  at  Level  1  and 
played  against  a  number  of  missions  in  the  O3  4  utility  mapping,  it  would  appear  that  we  have  proper 
measures  for  what  might  be  called  ballistic  survivability.  We  have  ignored  all  other  factors  in 
survivability  including  the  probability  of  detection  at  various  wavelengths,  agility,  etc.  One  can  think  of 
overall  survivability  as  the  combined  output  of  an  O3  4  utility  map,  using  not  only  the  Level  3  capability 
metrics  of  ballistic  measures  but  also  the  similar  capability  metrics  from  all  of  the  other  disciplines  which 
affect  survivability.  Further  discussion  of  the  live  fire  taxonomy  application  is  found  in  Deitz  (1996). 

2.2  Army  V/L  Modeling.  By  nature,  LFTs  are  empirical.  Therefore,  it  is  paramoimt  that  the  V/L 
model  used  for  simulation  be  capable  of  estimating  the  probability  of  occurrence  of  any  particular 
empirical  test  result.  Furthermore,  it  is  important  that  the  model  be  capable  of  calculating  data  in  all  of 
the  four  V/L  spaces.  Unfortunately,  many  models  give  deterministic  results  and  cannot  produce 
information  in  the  four  V/L  spaces. 

Two  Army  deterministic  models  have  been  used  in  the  past  Vulnerability  Analysis  Methodology 
Program  (VAMP)  and  Vulnerability  Analysis  for  Surface  Targets  (VAST).  The  LOF  and  component 
probability  of  kills  generated  by  these  models  are  expected  values.  The  results  of  any  particular 


Note  that  VAMP  is  only  capable  of  producing  LOFs. 
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empirical  test  may  or  may  not  match  these  expected  values.  Thus  it  is  necessary  to  use  a  model  that 
generates  entire  distributions  of  target  LOFs  and  damage  vectors,  allowing  the  estimation  of  the  probability 
of  occurrence  of  any  particular  empirical  test  result.  The  current  Army  model  used  for  simulating  LFT&E 
for  land  vehicles  is  called  the  Stochastic  Quantitative  Analysis  of  System  Hierarchies  (SQuASH). 

2.2.1  The  Army’s  SQuASH  Model.  The  SQuASH  model  (Ozolins  1988;  Deitz  and  Ozolins  1989) 
uses  repeated  sampling  of  random  variables  to  provide  highly  detailed  predictions.  A  distribution  of 
damage  vectors  where  each  vector  describes  the  functional/nonfunctional  state  of  every  critical  vehicle 
component  is  computed  by  repeatedly  simulating  the  threat  impact.  Component  damage  vectors  are  the 
only  necessary  and  sufficient  measure  for  analytically  verifying  agreement  between  a  model  and  an  LFT. 
Estimates  for  the  probability  of  occurrence  of  a  particular  target-threat  interaction  (Space  2  outcome)  can 
be  computed  from  the  predicted  distribution  of  damage  vectors.  By  incorporating  the  degraded  states 
vulnerability  methodology  (DSVM),  SQuASH  has  the  ability  to  measure  capability  degradation  in  terms 
of  capability  categories  (Space  3).  The  SQuASH  model  can  also  derive  the  vehicle  LOFs  distributions 
and  probability  of  catastrophic  kill.  Thus,  in  the  spaces  framework,  SQuASH  can  perform  the  Space  1 
to  2  mapping  (physics),  the  Space  2  to  3  mapping  (engineering  capability),  and  the  Space  3  to  4  mapping 
(battlefield  conditions/doctrine). 

Inputs  to  SQuASH  are  numerous,  covering  the  initial  conditions  in  Space  1  to  algorithms  for  Oj  2> 
O2  3,  and  O3  4  mappings.  Inputs  include  a  detailed  three-dimensional  geometric  characterization  of  the 
target  (e.g.,  components,  spatial  location  of  components,  component  properties,  etc.),  threat  information 
(e.g.,  shaped-charge  [SC]  waihead/jet  performance,  residual  penetration  capabilities),  the  threat’s  impact 
location(s)  (either  a  specific  location  or  a  set  of  Monte  Carlo  impact  locations  for  a  full  view),  spall  inputs 
(e.g.,  the  expected  number  of  lethal  fragments),  the  critical  component  probability  functions  of  a  kill  given 
a  hit,  deactivation  diagrams,  and  DSVM  fault  trees. 

All  critical  components  as  well  as  noncritical  components  capable  of  producing  spall  or  shielding  spall 
must  be  geometrically  modeled  to  accurately  predict  LFT  observables.  Deactivation  diagrams  from  the 
criticality  analysis  must  also  be  specified.  In  the  case  of  SQuASH  code  using  the  DSVM,  a  set  of  DSVM 
fault  trees  are  used  to  define  critical  components  and  their  relationship  to  each  of  the  capability  categories. 
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Monte  Carlo  simulations  are  performed  for  each  target-threat  interaction  (i.e.,  LFT  shot).  Monte  Carlo 
sampling  is  used  for  the  following  random  variables: 

•  The  threat’s  impact  location  on  the  target. 

•  The  penetration  capability  of  a  kinetic  energy  (KE)  munition  on  initial  target  impacts. 

•  The  residual  KE  penetration  ability. 

•  The  initial  penetration  depth  of  an  SC  jet. 

•  The  spatial  characterization  of  spall. 

•  The  number  of  lethal  spall  fragments. 

•  A  kill/no-kill  assessment  of  a  critical  component  from  the  effects  of  the  main  KE  penetrator,  KE 
penetrator  fragments),  or  an  SC  jet. 

•  The  effects  of  the  number  of  spall  fragments  impacting  the  component. 

SQuASH  output  for  repeated  sampling  at  a  given  threat  impact  location  includes  the  distribution  of 
damage  vectors  (representing  the  fimctional/nonfunctional  capabilities  of  all  critical  components),  the 
probabilities  of  perforating  armor,  the  distribution  of  residual  penetration,  and  the  probabilities  of  loll  for 
individual  components.  The  SQuASH  analysis  may  also  include  the  distribution  of  mobility  LOFs,  the 
distribution  of  firepower  LOFs,  and  the  distribution  of  vehicle  probability  of  catastrophic  kills.  Figure  3 
illustrates  SQuASH  in  the  space  framework. 

2.3  The  Testing  Process.  Models  and  analysis  methods  that  are  to  be  applied  in  support  of  LFT 
(premilestone  [PMS]  II)  should  include  information  at  all  stages  at  which  the  vulnerability  of  the  system 
can  be  affected.  Thus,  for  each  threat  damage  mechanism,  a  complete  set  of  vulnerability  information 
includes  component  vulnerability  data,  subsystem-component  interactions  (fault  trees),  and  subsystem 
interactions.  Additionally,  if  damage  mechanisms  are  not  independent,  combined  effects  for  components, 
subsystems,  and  multiple  subsystems  must  be  included.  For  whatever  type  of  model  is  applied,  it  is 


15 


SQuASHi^2 

stochastic 

variables 


Hit  point  variance 
Warf)ead  Performance 
Residual  Penetrator  Deflection 
Sf:«ll  Production 

Oornconerit  Pk/h  Characterization 


Caicuiation  of  Degradation 
Fault  Trees 


SQuASHz^s 

Full  Distribu¬ 
tion 


Remaining  Battlefielcl  LHiRty 


SQuASHs^ 
Full  Distribu¬ 
tion 


Figure  3.  Spaces  of  vulnerability  analysis — SOuASH  model. 

reasonable  to  assume  an  uncertainty  for  each  stage  of  input  as  well  as  an  uncertainty  each  time  the  model 
aggregates  damage  among  components,  within  subsystems,  and  between  subsystems  (to  the  system  level). 
The  uncertainty  associated  with  aggregation  is  due  to  possible  dependent  effects  among  various 
components  and  subsystems  or  among  various  damage  mechanisms  or  both. 

The  tradeoffs  between  fiill-up  LFT  and  other  information-gathering  plans  must  address  the  ability  to 
obtain  information  at  each  of  these  stages.  Component  testing  provides  only  component  vulnerability  data 
with  no  information  on  interaction  of  components.  Subsystem  testing  can  provide  information  on  both 
individual  components  and  synergistic  effects  on  interdependent  components,  but  provides  no  information 
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on  synergistic  effects  of  multiple  subsystems  (e.g.,  fire  initiation  due  to  interaction  of  sparking  from 
electrical  components  and  leakage  of  fuel  or  hydraulic  fluid).  Multiple  subsystem  tests  can  provide  data 
on  individual  components,  subsystems,  and  interaction  of  the  subsystems  tested.  Each  of  these  kinds  of 
test  are  usually  designed  to  investigate  specific  failure  modes,  interaction  of  failure  modes,  or  possibly 
interaction  of  damage  mechanisms.  Additionally,  tests  with  fewer  components  are  generally  less  expensive 
than  tests  farther  up  the  information  hierarchy. 

LFTs  have  the  potential  of  providing  the  most  information  per  shot  and  are  the  only  way  to  observe 
the  interaction  of  all  the  systems  to  multiple  synergistic  damage  mechanisms.  LFTs  are  either  engineering 
shots  or  random  shots.  Engineering  shots  are  selected  to  investigate  specific  system  changes,  such  as 
vulnerability  reduction  measures  (VRM),  or  to  address  other  issues.  Random  shots  are,  as  the  name 
implies,  selected  to  hit  the  target  at  a  random  location,  azimuth,  and  elevation.  Engineering  shots  could 
be  thought  of  as  designed  tests;  however,  the  small  sample  sizes  associated  with  LFT  and  the  absence  of 
replicated  shots  do  not  permit  good  inferences  on  the  issues.  Additionally,  the  full-up  LFT  does  not 
provide  a  controlled  environment  in  which  data  can  be  collected  at  the  appropriate  stages  or  in  quantities 
that  allow  high-confidence  inferences.  For  example,  an  LFT  provides  no  information  on  the 
characterization  of  behind-armor  debris  (BAD),  because  the  data  collection  techniques  are  invasive  to  the 
test  outcome.  Also,  since  each  of  the  LFTs  will  likely  impact  a  different  section  of  the  target,  many 
components  are  never  affected  by  the  shots  and  many  others  are  affected  by  only  a  very  few  shots. 
Nevertheless,  given  that  a  model  is  hypothesized  as  simulating  the  actual  LF  event,  it  takes  fewer  shots 
to  validate  the  model  than  it  would  to  create  the  model.  For  validation  of  the  simulation  over  many 
target-threat  combinations,  there  is  no  substitute  for  full-up  LFT  data.  Many  submodels,  such  as  BAD 
generation  can  be  validated  with  other  test  data;  however,  only  LF  data  provides  all  the  damage 
mechanisms  against  all  systems  as  well  as  their  synergistic  effects. 

An  additional  consideration  is  the  requirement  that,  for  LFT,  the  system  being  replaced  is  tested 
"side-by-side”  with  the  new  system.  Therefore,  the  cost  to  LFT  the  Paladin,  for  instance,  should  include 
the  cost  to  LFT  the  M109A2/A3  on  a  shot-by-shot  basis.  Obviously,  this  escalates  the  total  cost  of  LFT. 
Of  course,  if  the  system  has  an  entirely  new  mission,  then  this  is  not  a  factor.  Further  discussions  of  live 
fire  modeling  can  be  found  in  Deitz,  Saucier,  and  Baker  (1996)  and  m  Deitz  and  Saucier  (1996). 
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3.  CONNECTIVITY  OF  DAMAGE-TO-ENGINEERING  CAPABILITY  IN  THE  ACQUISITION 
PROCESS 

The  V/L  process  structure  provides  an  approach  that  applies  to  areas  other  than  V/L  analyses.  Recall, 
all  events,  through  the  determination  of  remaining  capabilities,  are  engineering  observables  or  measurables; 
that  is,  one  could  physically  observe  or  measure  these  phenomena  in  the  field.  Consequently,  the  process 
structure,  specifically,  the  use  of  fault  trees  to  describe  the  functionality  of  subsystems  and  capabilities, 
permits  analyses  across  the  spectrum  of  Army  concerns  for  a  combat  system  based  on  the  same  set  of 
required  capabilities.  This  increases  clarity  about  which  capabilities  are  important  and  provides  a  tool  for 
communication  among  the  analysis  community.  Several  such  applications  include:  (1)  comparison  of 
analytical  predictions  with  experimental  results;  (2)  battle  damage  repair  (BDR)  analyses;  (3)  RAM 
analyses;  (4)  C^erational  Requirements  Documents  (ORDs);  and  (5)  force-level  war  games. 

3.1  LFT  Assessment  Process.  An  important  application  of  the  V/L  process  structure  is  in  the 
comparison  of  analytical  data  with  results  of  actual  testing.  The  approach  promulgated  by  the  process 
structure  provides  an  excellent  means  by  which  to  evaluate  both  live  fire  experimental  data  and  potential 
shot  locations  prior  to  actual  firings.  Analytical  predictions  allow  the  evaluator  to  select  the  most 
meaningful  shots;  thus  shots  which  may  provide  minimal  data  can  be  eliminated,  or  shots  which  could 
result  in  catastrophic  loss  of  the  system  can  be  postponed  until  the  end  of  the  shot  series. 

Most  importantly,  a  combination  of  experiments  and  modeling  can  be  employed  to  maximize  the 
characterization  of  system  capability  while  minimizing  the  cost  of  such  characterization.  A  critical 
requirement  for  analytical  methodologies  to  be  used  in  this  manner  is  that  the  methodologies  parallel  test 
events  and  have  ouq)uts  which  are  measiurable  and/or  observable.  Validation  of  methodologies,  the  output 
of  which  can  only  be  inferred  indirectly  from  test  events,  is  at  best  extremely  difficult.  The  V/L  process 
structure  defines  an  appropriate  framework  for  such  methodologies. 

Traditional  combat  utility  metrics  that  infer  battlefield  effectiveness  directly  from  damaged  components 
suffer  from  two  major  difficulties.  First,  such  metrics  assume  no  variability  in  engineering  capability  from 
a  given  damage  vector,  test  experience  from  both  the  Joint  Live  Fire  (JLIO  and  the  Army  Live-Fire  (ALPO 
test  programs  has  shown  this  assumption  to  be  false.  Second,  since  these  traditional  measiures  constitute 
an  average  over  all  mission  and  scenario  factors,  they  provide  no  means  for  assessing  individual  subsystem 
capability  in  specific  contexts.  The  recent  conflict  in  Southwest  Asia  pointed  this  out  in  dramatic  fashion. 
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with  U.S.  armor  performing  at  levels  far  above  the  average  expectation.  Such  assessments  can  only  be 
done  at  Level  3.  The  use  of  fault  trees  allows  one  to  get  to  Level  3  in  the  V/L  process  as  well  as 
providing  the  crucial  interim  step  between  Levels  2  and  4.  Going  directly  from  Level  2  to  Level  4,  that 
is,  inferring  combat  utility  directly  from  system  damage  as  is  done  with  the  traditional  Standard  Damage 
Assessment  List  (SDAL),  does  not  provide  a  measurable  or  observable  output.  Thus,  one  cannot  relate 
the  damage  from  the  LFT  to  the  LOF  value  provided  by  the  SDAL  or  even  to  battlefield  utility;  there  is 
no  direct  comparison  between  the  two.  The  fault  trees,  in  tandem  with  the  V/L  process,  provide  the  means 
for  this  direct  comparison;  it  is  the  inclusion  of  Level  3  which  makes  this  possible. 

3.2  BDR.  Figure  4  depicts  the  analytical  paradigm  for  the  BDR  methodology,  laid  out  in  toms  of 
the  V/L  process  structure.  BDR,  or  any  kind  of  repair,  can  be  modeled  using  the  V/L  process  structure 
approach  in  the  following  manner.  Given  an  initial  set  of  component  damage  at  Level  2,  a  mapping  can 
be  performed  (using  the  DSVM)  to  determine  the  remaining  capabilities  of  the  system  at  Level  3.  This 
represents  the  capability  of  the  system  given  no  repair  is  performed.  If  one  can  establish  repair  priorities 
and  required  repair  times,  one  can  perform  a  sensitivity  analysis  to  determine  the  usefiilness  of  repairs  by 
attempting  to  do  whatever  repairs  are  possible  in  the  allotted  time.  This  provides  a  second  set  of  damaged 
components,  one  which  is  (possibly)  a  subset  of  the  original  set.  Using  this  new  damage  component 
vector  (Level  2),  a  mapping  is  performed  again  to  determine  the  remaining  capabilities  of  the  system  given 
the  affected  repairs.  After  a  comparison  is  made  between  the  original  set  of  remaining  capabilities  and 
the  new  set  resulting  from  repair,  an  assessment  can  be  made  of  the  usefulness  of  the  repairs  (i.e.,  what 
did  it  gain  the  system  in  terms  of  capabilities). 

A  series  of  repairs  can  be  identified,  and  sensitivity  analyses  performed,  to  determine  what  capabilities 
the  system  gains  as  a  result  of  varying  amounts  of  repair  time  and  parts  stockage.  These  analyses  can 
indicate  what  repairs  are  necessary  in  terms  of  system  performance,  what  types  of  spare  parts  need  to  be 
stocked,  and  what  the  critical  path  is  in  terms  of  needed  repair.  It  should  be  noted  that  when  the  repairs 
are  attempted  a  system  may  remain  at  the  same  damage  point  as  the  one  before  repairs  were  affected 
(i.e.,  not  enough  time  was  allocated).  Conversely,  a  system  may  be  returned  to  fully  functional  if  all  the 
damaged  components  were  repaired.  In  addition,  the  sensitivity  analyses  may  indicate  whether  or  not  the 
system  can  continue  a  certain  mission,  given  the  capabilities  required  are  available. 

Another  way  to  view  the  BDR  methodology  is  to  start  at  Level  4  and  ask  the  question,  what  is  the 
system’s  mission  and  what  capabilities  are  needed  to  accomplish  it?  This  returns  the  system  to  Level  3 
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to  determine  whether  or  not  the  capabilities  are  available.  If  not,  the  path  is  traced  back  to  Level  2  to 
determine  what  components  are  needed  to  permit  the  functioning  of  the  capabilities  required  to  accomplish 
the  mission;  this  gives  an  indication  of  what  components  need  to  be  repaired. 

Currently,  diere  is  no  uniform  treatment  of  repair  across  the  triservices  or  even  across  combat  arms 
within  the  Army.  The  BDR  methodology  described  here  solves  the  problem  and  provides  auditability  and 
consistency  of  results.  It  also  provides  an  approach  whereby  a  standard  accovmting  of  consequences,  in 
terms  of  performance  and  cost,  of  various  levels  of  repair  can  be  determined;  this  permits  uniform 
comparisons  across  the  mission  areas. 
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3.3  RAM  Analysis.  Another  area  of  application  for  the  process  structure  is  the  extension  of  fault  tree 
analysis  to  address  the  problem  of  RAM.  An  effort  is  underway  to  identify  the  commonalities  between 
the  vulnerability  and  reliability  analysis  techniques.  The  use  of  fault  tree  analysis  would  not  only  clarify 
the  capabilities  of  concern  for  the  combat  system  but  provide  the  same  starting  point  for  both  vulnerability 
and  RAM  analyses.  Although  the  damage  mechanisms  may  be  different,  the  effect  on  the  combat 
system’s  functionality  should  be  the  same  for  a  given  set  of  lost  components.  To  investigate  the  feasibility 
of  this  proposition,  the  Ballistic  Vulnerability/Lethality  Division  (BVLD)  formed  a  working  group  with 
the  AMSAA  Reliability,  Availability,  and  Mamtainability  Division  (RAMD)  to  address  problems  such  as 
familiarization  with  each  other’s  analytical  methods  and  determination  of  a  single  nomenclature  for  use 
in  both  V/L  and  RAM  analyses. 

The  initial  premise  of  this  effort  was  that  the  methods  for  assessing  the  damage  may  take  different 
approaches  but  should  yield  the  same  results.  Both  analyses  are  concerned  with  determining  remaining 
combat  system  capability  once  a  component  becomes  nonfunctional.  That  is,  both  must  assess  combat 
system  damage.  The  V/L  analyst  determines  the  lethality  of  a  weapon  or  the  vulnerability  of  a  combat 
system  in  terms  of  functional  damage;  the  RAM  analyst  determines  the  reliability  of  the  combat  system 
by  investigating  functional  failures.  To  do  this,  both  analysts  must  develop  an  understanding  of 
component  interrelationships  and  relate  these  components  to  the  system’s  required  capabilities.  While  the 
V/L  analyst  is  concerned  with  critical  components  (i.e,  components  required  to  allow  mission 
performance),  the  RAM  analyst  is  concerned  with  all  components.  For  example,  the  commander’s  seat 
is  noncritical  in  terms  of  mission  performance  for  a  V/L  analysis  and,  therefore,  is  not  considered  during 
the  analysis.  However,  a  RAM  analysis  needs  to  assess  the  commander’s  seat  in  order  to  determine 
whether  or  not  it  meets  its  reliability  criteria.  The  components  of  concern  for  the  V/L  analyst  are  a  subset 
of  those  of  concern  to  the  RAM  analyst.  Once  the  capability  levels  are  identified  and  the  fault  trees 
developed,  each  analyst  could  select  those  fault  trees  applying  to  his/her  analysis. 

The  objective  of  the  initial  BVLD/RAMD  investigation  was  to  determine  if  the  two  methods  yielded 
the  same  results  and  to  investigate  the  differences  and  similarities  in  order  to  determine  where  the  two 
processes  could  be  combined.  The  initial  effort  established  the  functional  loss  of  the  MlAl  for  a  given 
set  of  kOled  components,  selected  from  a  recent  vulnerability  analysis.  This  set  of  killed  components  was 
used  by  both  reliability  and  vulnerability  analysts  to  assess  which  functions  on  the  vehicle  were  affected. 
Each  used  his  normal  procedures.  For  the  vulnerability  analyst,  the  criticality  analysis  of  the  MlAl  was 
consulted.  The  reliability  analyst  made  use  of  the  RAMD  FaUiue  Criteria  Document  for  the  Ml  El  tank 
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and  conversations  with  the  MlAl  project  manager.  The  RAMD  failure  criteria  use  block  diagrams  which 
group  components  by  function  (e.g.,  mobility,  fire  control,  etc.).  This  allows  the  analyst  to  show  the 
relationship  between  the  function  and  the  components  that  make  up  the  function.  A  description  is  then 
developed  for  the  block  diagram,  which  contains  narrative  representations  of  the  basic  functions  followed 
by  failure  modes  of  the  hardware  associated  with  the  function.  This  description  reports  which  components 
cause  functional  loss,  functional  degradation,  or  have  no  effect  at  all. 

Once  the  individual  analyses  were  completed,  the  results  were  compared.  Although  there  appeared 
to  be,  in  general,  no  disagreement  in  assessed  functional  loss,  the  different  processes  and  nomenclature 
made  it  difficult  to  be  certain.  The  reliability  assessment  provided  more  specific  information  on  functional 
loss,  but  required  more  effort  than  the  vulnerability  approach.  Generally,  the  comparisons  were  fairly 
straightforward,  however,  some  difficulties  did  arise.  For  instance,  some  components  contributed  to  more 
than  one  function  and  thus  were  described  in  more  than  one  RAM  narrative  section,  depending  upon  the 
function  being  described.  Unless  the  RAM  analyst  knew  this  a  priori,  not  all  loss  functions  may  be 
identified.  A  second,  more  common  problem,  dealt  with  matching  functions  selected  by  the  analyst  when 
developing  their  criteria.  In  some  cases,  it  was  unclear  if  the  two  analysts  were  describing  the  same 
functions. 

A  single  set  of  combat  system  capabilities  should  be  identified  early  in  the  developmental  cycle  for 
use  in  all  system  analyses,  including  VL,  RAM,  and  logistics.  Techniques  employed  have  not  been 
consistent  across  organizations,  most  likely  due  to  a  lack  of  realization  by  the  various  analysts/project 
managers  of  the  inherent  similarities,  resulting  in  the  development  of  different  capability  requirements  for 
different  applications.  Developing  the  list  of  required  capabilities  early  in  the  cycle  would  avoid  the 
aforementioned  comparison  problems  and  provide  the  basis  for  all  subsequent  analyses  of  the  combat 
system. 

These  examples  indicate  that  the  use  of  fault  tree  analysis  could  make  the  RAM  process  easier  and 
faster.  Additionally,  if  fault  trees  for  a  given  combat  system  were  developed  jointly  by  ARL  and  AMSAA 
analysts,  early  in  the  analytical  process,  some  of  the  differences  in  answers  could  be  avoided,  or  more 
easily  accounted.  As  a  result,  BVLD  and  RAMD  agreed  to  a  joint  effort  aimed  at  developing  common 
standards  and  practices  for  the  identification  of  required  combat  system  capabilities,  the  components  that 
contribute  to  each  capability,  and  the  interrelationship  of  these  components  to  overall  system  functionality 
and  impact  when  the  component  (or  subassembly)  is  lost  or  partially  lost.  This  effort  is  intended  to 
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develop  a  standard  technique  for  use  by  all  organizations.  Development  of  this  technique  will  enhance 
the  Army’s  ability  to  provide  detailed  and  consistent  system  capability  requirements  across  the  spectrum 
of  analyses.  In  addition,  savings  can  be  identified  in  terms  of  both  time  and  money  by  the  reduction  of 
duplication  of  effort.  It  is  anticipated  that  this  effort  will  be  the  starting  point  for  all  subsequent  combat 
system  analyses  to  include  vulnerability,  RAM,  logistics,  survivability,  and  effectiveness. 

3.4  ORDs.  An  additional  application  of  fault  trees  is  the  determination  of  capability  criteria  for 
ORDs.  Per  DODI  5000.2,  an  ORD  is  a  formatted  statement  containing  performance  and  related 
operational  parameters  for  a  proposed  concept  or  system.  The  initial  ORD  describes  each  concept 
proposed  at  MS  I  to  include  terms  of  minimum  acceptable  requirements  that  define  the  system  capabilities 
needed  to  satisfy  the  Mission  Need  Statement  (MNS).  The  ORD  is  updated  and  expanded  for  MS  II  to 
include  thresholds  and  objectives  for  more  detailed  and  refined  functional  capabilities  and  characteristics. 
These  updates  are  based  on  the  results  of  the  tradeoff  studies  and  testing  conducted  during  Phase  I 
(Demonstration  and  Validation).  The  ORD  is  then  used  to  develop  the  system’s  requirements  for  contract 
specification  through  each  acquisition  cycle. 

One  of  the  major  objectives  of  the  ORD  is  to  define  the  required  functional  capabilities  and 
requirements.  These  definitions  are  to  include  an  objective  which  represents  a  measiurable,  beneficial 
increase  in  capability  or  operations  and  supports  the  minimum  acceptable  level  specified  in  the  document. 
Historically,  though,  these  objectives  have  been  defined  in  terms,  usually  probability  of  kill  (Pk),  which 
are  both  physically  unmeasurable  and  vague  in  quantification.  Because  of  these  inherent  problems, 
definition  of  the  requirements  in  terms  of  the  system’s  required  capabilities  (at  Level  3  in  the  V/L  process 
structure)  would  provide  well-defined,  measurable  objectives.  As  an  example,  the  application  of  the 
DSVM  approach  to  the  Advanced  Field  Artillery  System  (AFAS)  will  be  discussed.  This  work  was 
initially  performed  at  the  direction  of  the  Program  and  Vulnerability  Assessment  Office  of  the  Assistant 
Secretary  of  the  Army  for  Research,  Development,  and  Acquisition  (SARDA).  (It  should  be  noted  that 
this  approach  works  just  as  well  for  lethality  [i.e.,  missiles]  as  it  does  for  vulnerability/survivabUity  of 
combat  systems.  Thus,  the  use  of  the  term  "combat  system"  implies  any  system  which  can  be  described 
in  terms  of  required  capabilities.) 

The  AFAS  is  the  future  self-propelled  howitzer,  designed  as  die  replacement  for  the  M109A6  Paladin. 
The  proposed  ORD  for  AFAS  was  reviewed  with  selected  functional  requirements  rewritten  in  terms  of 
required  capabilities.  As  neither  the  firepower  nor  the  mobility  requirements  were  described  in  terms  of 
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physically  measurable  quantities,  suggestions  were  made  as  to  how  these  requirements  could  be  rewritten. 
Firepower,  for  example,  could  be  expressed  in  terms  of  lower  and  upper  bounds  of  an  acceptable  level 
as  shown  in  Table  1. 


Table  1.  Example  Firepower  Requirements 


Requirement 

Rate  of  Fire 

Acceptable  Boimd 

Upper 

at  least  12  rd/min  for  5  min 

Lower 

not  less  than  6  rd/min  for  5  min 

The  same  type  of  physical,  or  engineering,  metric  could  be  applied  to  the  mobility  requirement.  For 
example,  speed  could  be  defined  in  terms  of  the  required  (and  desired)  level  for  different  environments 
(i.e.,  at  least  25  mph  in  European  rolling  hills  or  40  mph  in  Southwest  Asian  desert). 

The  quantification  of  the  requirement  in  terms  of  engineering  metrics  permits  easier  evaluation  as  to 
whether  or  not  the  requirement  has  been  met.  The  benefits  of  this  approach  are  numerous.  First,  the 
requirements  are  expressed  in  terms  of  capabilities  which  can  be  explicitly  measured  and  which  are 
separated  into  the  different  capability  categories  (i.e.,  mobility,  firepower,  and  acquisition).  It  provides 
a  means  for  the  user  to  prioritize  the  capabilities  most  worth  preserving  as  well  as  provides  greater  insight 
into  the  military  utility  of  the  system  and  the  effectiveness  of  each  of  its  subsystems.  Most  importantly, 
it  provides  greater  clarity  as  the  user  and  developer  of  the  system  discuss  tradeoffs  between  what 
capability  is  wanted  vs.  what  can  be  affordably  built. 

3.5  Force  Level  War  Games.  The  modeling  of  system  functionality  with  fault  trees,  as  a  part  of  the 
mapping  from  Level  2  to  Level  3  in  the  V/L  process  structure  also  has  consequences  for  force-level 
modeling.  It  permits  a  more  accurate  portrayal  of  the  system’s  remaining  capabilities  as  a  result  of 
nonfunctional  components,  either  through  combat  damage  or  failure.  As  currently  modeled,  the  system 
is  either  fully  functional  or  fully  nonfunctional.  A  more  realistic  approach  would  be  to  model  the  system 
as  still  functional  but  in  a  degraded  operational  mode.  This  would  result  in  a  system  remaining  in  the 
engagement  longer  and  possibly  affecting  the  outcome  of  the  game.  In  addition,  information  from 
Level  2,  component  damage,  would  provide  detailed  information  on  damaged  parts,  thus  providing  more 
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accurate  data  on  spare  parts  requirements  and  the  need  for  BDR  (as  discussed  in  the  previous  section  on 
BDR). 

A  current  effort  underway  between  the  BVLD  and  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC)  Analysis  Center  at  White  Sands  Missile  Range  (TRAC-WSMR)  is  investigating  the  inclusion 
of  DSVM  metrics  in  the  CASTFOREM  force-level  model.  Specifically,  the  BVLD  and  TRAC-WSMR 
are  modifying  the  CASTFOREM  model  to  accept  the  DSVM  metrics  as  well  as  store  the  damaged 
component  vectors  available  from  the  Oj  2  mapping.  The  inclusion  of  the  DSVM  metrics  and  component 
information  in  this  model  wUl  allow  RAM  and  BDR  to  be  played  directly  in  the  models  and  their  effects 
on  the  systems  and  battle  outcome  to  be  directly  measured. 

3-6  Summary  Connectivity  of  Damage-to-Engineering  Performance  in  the  Acquisition  Process.  Both 
fault  tree  analysis  and  the  general  V/L  process  structure  have  applications  in  areas  outside  the  realm  of 
V/L,  as  evidenced  by  the  discussions  within  this  section.  The  process  structure  allows  the  V/L  models 
to  closely  parallel  LFT&E  events,  thus  providing  calibration  points  for  these  models.  Further,  the  ability 
to  clarify  the  operational  requirements  of  a  system  by  identifying  its  required  functions  provides  a  sound 
basis  for  communications  between  the  user  and  the  developer  throughout  the  acquisition  life  cycle  of  the 
system.  The  same  analytical  approach  can  be  applied  to  RAM  and  BDR  analysis,  which  allows  one  to 
evaluate  the  major  aspects  of  the  acquisition  cycle  using  the  same  process.  Finally,  the  inclusion  of  the 
damage  vector  information  and  the  DSVM  metrics  in  force-level  models,  such  as  CASTFOREM,  will 
allow  the  major  aspects  of  the  acquisition  cycle  to  be  played  simultaneously.  The  overall  improvement 
in  communications  and  analyses,  as  provided  by  the  V/L  process  structure,  can  only  improve  the  Army’s 
ability  to  provide  timely,  credible  analyses. 

4.  THE  ROLE  OF  LIVE-FIRE  TESTING  (LFT) 

4.1  Department  of  Defense  Reeulation  5000.2-R. 

DOD  5000.2-R  requires  test  and  evaluation  programs  to  be  structured  to  integrate  all  developmental, 
operational,  and  LFT&Es  as  well  as  modeling  and  simulation  activities  into  an  effective  continuum 
throughout  the  acquisition  cycle  for  applicable  combat  systems.  In  fact,  DOD  5000.2-R  requires  test  and 
evaluation  planning  to  begin  in  Phase  0,  Concept  Exploration,  of  the  acquisition  cycle. 
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Figure  5  indicates  where  in  the  &st  three  phases  of  the  acquisition  cycle  the  most  significant  events 
regarding  survivability  occur  as  well  as  their  linkage  to  program  technical  initiatives  and  to  the  maturation 
of  system  technical  attributes. 

Conventional  ballistic  vulnerability  reduction  concerns  the  response  of  the  target  vehicle  to  an 
interaction  with  a  threat  munition  and  is  a  major  component  of  survivability.  The  success  and 
effectiveness  of  the  vulnerability  reduction  effort  is  obviously  related  to  the  choices  of  technical  solutions 
to  mission  needs  identified  in  the  initial  Materiel  Need  Statement,  the  maturation  of  these  technologies, 
and  their  incorporation  into  system  design  during  Engineering  and  Manufacturing  Development.  A 
comprehensive  vulnerability  reduction,  analysis,  and  experimentation  program,  though,  is  required  to 
ensure  a  successful  transition  from  concept  to  production  insofar  as  the  selection,  development, 
optimization,  and  integration  of  vulnerability  reduction  alternatives  into  the  final  system  design.  Systems 
engineering,  for  example,  is  a  management  process  espoused  in  DOD  5000.2-R  to  translate  operational 
needs  into  a  configured  system  through  a  systematic  integration  of  technical  inputs  from  the  entire 
development  community.  Risks  are  characterized  through  early  testing  and  demonstrations  of  system 
elements.  The  ultimate  objective  of  the  systems  engineering  process  is  to  ensure  and  verify  that  the 
system  design  meets  the  operational  needs. 

While  DOD  5000.2-R  requires  an  assessment  of  the  survivability  to  all  threats  found  in  the  various 
levels  of  conflict  prior  to  Milestone  n  and,  in  a  general  sense,  vulnerability  testing  relatively  early  in  the 
acquisition  cycle,  the  only  specific  vulnerability  testing  program  identified  is  the  LFT,  which,  if  conducted, 
occurs  immediately  before  full-scale  production. 

Congress  has  mandated  in  Title  10,  United  States  Code,  Section  2366,  that  ACAT I  and  II  programs 
for  covered  systems  (vehicles,  weapon  platforms,  or  conventional  weapon  systems  designed  to  provide 
some  degree  of  protection  to  the  user  in  combat)  must  include  provisions  for  vulnerability  LFT.  Such 
testing  is  intended  to  demonstrate  and,  in  conjunction  with  modeling,  determine  the  vulnerability 
characteristics  of  the  candidate  system  to  the  spectrum  of  threats  likely  to  be  encountered  in  combat.  The 
statute  provides  a  waiver  if  it  can  be  demonstrated  that  LFT  would  be  prohibitively  expulsive  or 
impractical  and  it  can  be  shown  that  the  survivability  of  the  system  can  be  evaluated  in  other  ways.  Such 
a  waiver  must  be  made  prior  to  the  beginning  of  the  Engineering  and  Manufacturing  Development  phase. 

Since  LFT  is  the  only  vulnerability  testing  specifically  identified  in  DOD  5000.2-R,  it  may  be 
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Figure  5.  Acquisition  process. 


appealing  to  defer  experiments,  testing,  and  analysis  until  the  time  LFT  is  to  be  conducted.  However,  LFT 
should,  in  actuality,  be  the  culmination  of  a  much  more  comprehensive  and  long-term  vulnerability 
reduction,  analysis,  and  experimentation  program  that  includes  not  only  the  formal  LFT  but  all 
vulnerability  analyses,  experiments,  and  testing  conducted  from  concept  exploration  through  low  rate 
production.  Relying  on  only  LFT  to  address  system  vulnerability  not  only  places  undue  risk  in  the 
program  but  practically  ensures  the  system  will  exhibit  less  than  optimum  vulnerability  characteristics. 
A  well-formulated  vulnerability  reduction,  analysis,  and  experimentation  program  initiated  early  and 
structured  in  a  budding-block  manner  will  ensure  more  robust  system  survivability  and  can  reduce  or  even 
eliminate  the  need  for  expensive  full-up  LFT  just  prior  to  full-rate  production. 

4.2  Viilnftrahilitv  Reduction.  Analysis,  and  Experimentation  Program  Structure 

The  vulnerability  reduction,  analysis,  and  experimentation  program  must  address  system  survivability 
from  three  points  of  view.  First,  it  must  ensure  requirements  for  vulnerability  reduction  evolving  from 
the  initial  Materiel  Need  Statement  are  reasonable,  achievable,  and  practicable.  The  whole  battlefield 
threat  likely  to  be  encountered  must  be  addressed.  System  survivability  must  not  depend  on  technologies 
that  caimot  be  reasonably  expected  to  mature  during  the  acquisition  process  or  which  have  an  unacceptable 
risk  of  doing  so.  On  the  other  hand,  requirements  should  be  aggressive  enough  to  take  advantage  of,  or 
even  challenge,  existing  applicable  technologies.  Incorporation  of  vulnerability  reduction  features  into 
system  design  must  not  unduly  burden  other  design  parameters.  Even  more  importantly,  they  must  not 
sacrifice  fightability. 

Second,  the  vulnerability  reduction,  analysis,  and  experimentation  program  must  be  formulated  to  help 
the  materiel  developer  ensure  that  design  requirements  are  being  met  as  the  acquisition  cycle  progresses. 
But,  even  beyond  requirements,  the  program  must  stimulate  measures  that  can  and  should  be  taken  to 
optimize  vulnerability  reduction  against  threats  or  conditions  beyond  the  actual  design  requirements.  The 
focus  here  is  on  applying  good  engineering  principles  to  reduce  vulnerability  and  ensure  the  design  is 
relatively  insensitive  to  at  least  minor  variations  in  threat  characteristics  with  minimal  additional  design 
cost. 

Third,  the  vulnerability  reduction,  analysis,  and  experimentation  program  should  be  structured  to 
reduce  program  risk  by  ensuring  successful  completion  of  LFT.  These  tests  are  conducted  on  production 
hardware,  which  means  they  are  conducted  very  late  in  the  development  cycle.  Clearly,  problems 
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identified  during  LFT  will  be  costly  to  remedy  and  will  most  certainly  at  least  delay  the  MS  IE  full-rate 
production  decision.  In  an  aggressive  vulnerability  reduction,  analysis,  and  experimentation  program, 
problem  areas  will  be  identified  and  addressed  long  before  the  design  is  Hnalized  and  in  time  to  effect 
appropriate  corrective  measures. 

The  vulnerability  reduction,  analysis,  and  experimentation  program  should  be  structured  to  progress 
in  a  manner  that  allows  all  efforts  to  take  advantage  of  and  build  on  previous  efforts.  Procedurally,  a 
combination  of  analytical  modeling,  experiments,  and  analysis  methodology  development  is  necessary. 
Modeling  progresses  from  low-resolution  analyses  of  design  concepts  early  in  the  acquisition  cycle  to 
high-resolution  modeling  to  support  LFT  near  production.  Experiments  are  conducted  early  to  characterize 
the  ballistic  performance  of  new  materials  or  design  concepts.  As  the  system  matures,  the  experiments 
on  design  concept  components  and  even  on  prototype  components  become  more  extensive  in  an  attempt 
to  characterize  the  vulnerability  of  what  will  eventually  be  the  actual  design.  Eventually,  laboratory 
experiments  give  way  to  field  testing  culminating  with  the  LFT  just  before  full-rate  production.  Damage 
mechanisms  and  component/subsystem  failure  modes  that  cannot  be  adequately  modeled  with  extant 
methodologies  must  be  addressed  in  methodology  development  efforts. 

Neither  the  analyses  nor  the  experiments  and  testing  are  conducted  in  a  vacuum.  Experimental  and 
test  results  are  used  to  formulate  inputs  to  the  modeling  codes.  Conversely,  analytical  results  provide 
valuable  guidance  to  experimental  design.  Figure  5  shows  the  types  of  experimental  and  analytical 
activities  required  throughout  the  acquisition  process  and  suggests  the  way  these  activities  are  linked  to 
provide  a  coordinated  comprehensive  vulnerability  reduction,  analysis,  and  experimentation  program.  The 
following  comments  discuss  these  activities  and  indicate  the  purpose  and  granularity  of  each  activity  as 
the  design  and  development  proceeds. 

In  the  early  stages  of  the  acquisition  process,  the  primary  focus  with  regard  to  vulnerability  reduction, 
analysis,  and  experimentation  is  on  low-resolution  modeling.  Because  only  concepts  are  being  considered 
at  this  stage,  there  is  not  enough  system  design  detail  to  permit  high-resolution  modeling.  In  fact,  what 
is  needed  at  this  stage  are  analyses  to  define  the  potential  of  competing  concepts  to  reduce  vulnerability 
and  to  define  the  limits  or  bounds  of  vulnerability  reduction  that  may  be  achieved  if  a  given  concept  is 
selected  for  system  development.  Ancillary  analyses  may  be  performed  to  assist  in  the  preparation  of  the 
ORD.  It  is  essential  that  the  ORD  contain  vulnerability  reduction  requirements  that  are  appropriate  given 
the  projected  threat  environment  and  usage  of  the  objective  system  but  at  the  same  time  are  achievable 
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with  available  technology  without  putting  an  undue  burden  on  other  design  parameters.  Again,  since 
design  details  of  competing  concepts  are  usually  sparse  at  this  point,  the  analyses  attempt  to  capture  the 
potential  (or  lack  of  potential)  of  major  vulnerability  reduction  technologies  known  to  be  seriously 
considered.  The  aim  is  not  to  quantify,  in  absolute  terms,  the  vulnerability  of  any  system  but  to  ensure 
the  proper  rank  ordering  of  competing  concepts. 

Because  of  schedule  availability,  this  is  an  optimum  time  to  conduct  a  high-resolution  analysis  of  the 
system  being  replaced,  if  that  is  the  case,  or  of  the  system(s)  being  supplemented  by  the  new  system.  This 
will  establish  a  baseline  quantification  of  vulnerability  that  wUl  be  used  later  in  the  acquisition  cycle  to 
gauge  the  improvement  achieved  in  the  new  system.  (Care  must  be  taken,  though,  to  resist  comparing 
low-resolution  model  analyses  of  concepts  with  the  high-resolution  analysis  of  existing  system(s)  because 
the  basis  of  comparison,  namely  relative  target  description  detail,  will  be  faulty.) 

Experiments  should  (or  in  some  cases  must)  be  conducted  to  get  a  handle  on  the  vulnerability 
reduction  characteristics  of  emerging  technologies.  Examples  of  such  experimental  efforts  include 
characterizing  the  ballistic  performance  of  composite  or  other  new  materials,  determining  the  impact 
sensitivity  of  stowed  munition  energetic  components,  and  identifying  likely  failure  modes  for  sensitive 
electronic  components.  It  is  not  usually  necessary  that  these  experimental  investigations  be  extensive  since 
there  is  no  assurance  that  investigated  technologies  will  be  pursued  in  development  (at  least  at  the  detail 
level,  composite  armor  might  be  ultimately  used,  but  the  specific  armor  may  not  be  known  at  this  point, 
for  example).  However,  it  is  often  necessary  to  conduct  adequate  experimentation  to  generate  rules  of 
thumb  that  will  boxmd  ballistic  performance,  provide  insights  into  potential  failure  modes,  and  guide 
technology  integration.  For  example,  it  would  probably  not  be  necessary  to  generate  new  penetration 
equations  for  all  new  types  of  materials  but  it  would  be  necessary  to  conduct  enough  experiments  to 
calibrate  existing  equations  for  similar  materials  to  be  able  to  predict  approximate  penetration  performance. 

There  may  be  a  need  to  begin  to  improve  or  even  extend  analysis  methodologies  at  this  point  if  it 
appears  that  extant  methodologies  will  be  deficient  later  in  the  program.  Whether  this  is  desirable  or  even 
possible  will  depend  on  the  likelihood  of  various  technologies  being  pursued  later  in  system  development. 
This  underscores  the  importance  of  a  comprehensive  and  continuing  experimental  program  begiiming  at 
the  earliest  stages  of  the  program. 

During  the  latter  part  of  Phase  0,  Concept  Exploration,  and  during  Phase  1,  Program  Definition  and 
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Risk  Reduction,  both  analyses  and  experiments  become  much  more  system  specific  than  they  were  earlier. 
But,  there  may  still  be  many  gaps  in  system  design  details  and  tradeoffs  will  continue  to  be  conducted  to 
define  the  system  configuration.  Low-resolution  analyses,  and  even  high-resolution  analyses  of  existing 
prototypes  in  many  cases,  should  begin  to  focus  on  specific  questions  and  issues  related  to  system  design. 
The  ORD  contains  specific  system  requirements  that  can  also  be  addressed.  One  important  task  is  to 
recast  ORD  requirements  that  are  typically  stated  in  terms  of  taxonomy  space  4  metrics  (loss  of  combat 
capability)  into  a  suite  of  system  performance  metrics  (space  3).  These  metrics,  which  can  be  tested,  are 
valuable  for  characterizing  the  vulnerability  of  the  system  in  terms  that  can,  with  follow-on  effectiveness 
modeling,  truly  address  siuvivability  in  several  different  mission  scenarios. 

Later  as  prototypes  mature,  it  is  possible  to  critique  emerging  designs  and  apply  engineering  design 
principles  that  foster  vulnerability  reduction.  Examples  include  using  redundancy  for  hydraulic,  fuel,  and 
electrical  circuits;  locating  critical  components  to  use  protection  afforded  by  heavy  structures;  and 
determining  optimum  armor  coverage. 

In  the  same  manner,  experiments  can  be  conducted  to  characterize  component  vulnerability,  including 
identification  of  significant  damage  mechanisms  and  failure  modes.  Experimental  efforts  will  focus  on 
space  2  metrics  and  space  1—2  mapping,  although  consideration  of  space  3  metrics  and  consequent 
space  2—3  mapping  should  also  begin.  For  armor  systems,  BAD  characterizations  are  usually  necessary. 
The  scope  and  complexity  of  these  experiments  will  vary  depending  on  hardware  availability  and  design 
maturity.  It  is  desirable  for  these  experiments  to  be  as  thorough  as  possible  at  this  point,  because  the  data 
will  be  required  for  subsequent  analyses  and  will  also  be  useful  for  detailed  design  in  the  Engineering  and 
Manufacturing  Development  phase.  It  may  be  desirable  to  conduct  limited  subsystem  level  tests  at  this 
time.  Examples  would  be  ballistic  hull/turret  tests  of  armor  vehicle  designs  to  determine  BAD 
distributions  and  quantify  the  potential  of  mitigating  schemes. 

Vulnerability  analysis  model  development  should  now  focus  on  continuing  development  of  algorithms 
to  account  for  new  damage  mechanisms  and  failure  modes  or  to  model  new  system  technologies.  The 
definition  of  degraded  states  begun  previously  should  be  refined  to  reflect  maturity  in  both  system  design 
and  operational  deployment  concepts. 

As  the  definition  of  the  system  matures,  it  is  both  possible  and  necessary  to  conduct  more  definitive 
analyses  and  more  comprehensive  experiments  and  tests.  Beginning  early  in  Phase  2,  Engineering  and 
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Manufacturing  Development,  high-resolution  modeling  is  used  to  help  system  designers  choose  among 
vulnerability  reduction  alternatives.  These  analyses  define  the  potential  payoff  of  competing  alternatives 
as  well  as  some  of  their  costs,  particularly  space  and  weight.  Once  specific  design  concepts  have  been 
chosen,  further  analyses  can  be  used  to  generate  data  to  help  perfect  the  design.  Both  high-resolution 
modeling  and  the  continued  application  of  vulnerability  reduction  design  principles  are  necessary  to 
accomplish  this  objective.  Even  though  system  design  is  now  focused  on  meeting  specific  design 
requirements,  including  those  for  vulnerability  reduction,  high-resolution  modeling  affords  an  excellent 
opportunity  to  analyze  the  vulnerability  of  the  system  to  the  spectrum  of  battlefield  threats.  It  will  usually 
become  apparent  that  some  threats  can  be  easily  defeated  or  tolerated  if  the  design  requirements  are  met. 
Others  will  be  shown  to  be  overmatches,  which  may  be  impractical  or  impossible  to  totally  defeat.  But, 
it  may  be  possible  to  mitigate  the  effects  of  these  threats  under  at  least  some  combat  conditions  through 
the  judicious  application  of  vulnerability  reduction  engineering  principles.  Often,  significant  gains  in 
protection  can  be  achieved  at  little  additional  cost.  It  is  not  the  purpose  of  modeling  to  design  the  system 
but  to  generate  data  through  which  the  best  design  can  evolve. 

As  with  analyses,  this  is  the  opportune  time  to  conduct  experiments  to  further  develop  component 
failure  modes  and  gain  better  insights  into  the  threat  damage  mechanisms  to  which  the  design  is 
particularly  susceptible.  Hopefully,  the  experiments  conducted  to  this  point  will  have  produced  the  basis 
for  continued  experiments  focusing  more  on  subsystem  and  system  attributes.  As  the  system  design 
matures,  it  is  possible  and  affordable  to  conduct  a  much  more  comprehensive  set  of  experiments  to 
investigate  and  quantify  these  phenomena.  Subsystem  level,  and  later,  system  level  experiments  and  field 
tests  should  be  conducted  to  identify  and  investigate  synergisms  that  cannot  be  thoroughly  addressed  at 
the  component  level.  While  many  of  these  synergisms  can  be  anticipated  quite  early,  even  in  preceding 
phases,  experiments  and  tests  at  this  point  can  be  conducted  in  a  manner  that  will  foster  the  identification 
of  unanticipated  synergisms.  Hiis  is  accomplished  by  including  as  many  components  in  the  test  assembly 
as  possible  and  by  instrumenting  for  data  that  otherwise  might  not  be  deemed  necessary.  Subsystem  tests 
are  also  necessary  to  fully  establish  selected  vulnerability  reduction  performance  limits.  For  example,  a 
full  suite  of  production  armor  moimted  on  a  complete  chassis  may  behave  far  differently  flian  subscale 
laboratory  coupons  in  a  laboratory  fixture.  Development  and  operational  tests  (DT  and  OT)  should  be 
exploited  when  possible  to  both  add  to  the  test  data  base  and  to  reduce  the  amount  of  full-up  vulnerability 
LFTs.  In  this  way,  costs  can  be  reduced  through  "piggy  backing"  onto  the  DT  or  OT  rather  than 
conducting  dedicated  vulnerability  tests.  For  example,  it  is  common  to  conduct  DT  on  armor  vehicles  to 
ensure  the  armor  installation  and  protection  meet  design  requirements.  This  series  could  easily  be 
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expanded  to  fire  overmatching  threat  mimitions  to  determine  the  upper  limits  of  protection  and  include 
at  least  selected  operational  subsystems  to  determine  their  response. 

The  "ultimate"  vulnerability  test  series  is  the  LFT  mandated  for  most  combat  systems.  These  firings 
are  to  be  conducted  on  production  quality  hardware  with  actual  threat  munitions,  or  at  least  with  approved 
surrogates  when  actual  munitions  are  not  available  or  are  prohibitively  expensive.  As  valuable  and 
revealing  as  these  full-up  tests  are,  they  are  no  substitute  for  coupon,  component,  and  subsystem 
experiments  conducted  throughout  the  acquisition  process.  Full-up  tests  simply  cannot  produce  the  type 
and  quantity  of  data  needed  to  develop  a  detailed  understanding  of  the  underlying  phenomena.  This 
further  emphasizes  the  need  for  a  comprehensive  vulnerability  reduction,  analysis,  and  experimentation 
program  throughout  the  acquisition  cycle. 

Both  modeling  and  component/subsystem  level  experiments  support  LFT.  Modeling  is  used  to  help 
select  shot  lines  and  also  (particularly  high-resolution  modeling)  to  predict  test  outcomes  (preshot 
predictions).  The  experiments  leading  up  to  LFT  provide  input  data  for  modeling  (component  kill 
probabilities,  failure  modes,  limits  of  redundancies,  etc.)  and  also  reduce  the  likelihood  of  unpleasant 
surprises  during  the  full-up  LFT. 

Just  because  a  system  has  undergone  LFT,  is  in  production,  and  has  been  fielded,  survivability  caiuiot 
be  neglected.  Seldom  will  a  system  be  fielded  and  retired  without  requiring  design  changes  to  address 
design  deficiencies  identified  after  production  or  to  reflect  changes  in  operational  use  or  threat  escalation. 
Advances  in  materials,  design,  and  manufacturing  technology  also  lead  to  design  changes  to  reduce  costs, 
improve  performance,  or  add  capabilities  to  the  system. 

Design  changes  for  any  of  these  reasons  can  affect  system  survivability.  In  fact,  the  LF  legislation 
requires  LFT  for  product  improvements  if  the  vulnerability  of  the  system  is  potentially  affected.  The  roles 
of  analysis  and  experimentation  are  essentially  the  same  as  they  are  during  system  design  except  that 
usually  it  is  possible  to  conduct  more  high-resolution  modeling  since  the  baseline  system  is  well  defined 
and,  hopefully,  well  characterized.  This  allows  the  effect  of  specific  design  proposals  to  be  thoroughly 
evaluated. 
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5.  COST-BENEFIT  OF  FULL-UP  LFT 


TTie  purpose  of  this  chapter  is  to  investigate  various  methodologies  which  would  quantify  the  costs, 
risks,  and  benefits  associated  with  full-up  LFT  and  with  various  alternative  testing  strategies.  Such  a 
methodology  should  serve  as  a  formal  procedure  for  the  request  of  a  waiver  from  full-up  LFT  of  a  system. 
Four  methodologies  have  been  proposed,  three  by  the  Army  (sections  5.1  and  5.2  provided  by  the 
ARL/Survivability/Lethality  Analysis  Directorate  (SLAD)/BVLD,  section  5.5  provided  by  University  of 
Wisconsin  by  DOD  contract)  and  one  (section  5.4)  by  the  U.S.  Air  Force  (USAF).  As  might  be  expected, 
they  all  incorporate  considerable  subjectivity,  and  they  vary  considerably  in  the  total  effort  needed  for 
implementation.  There  are  tradeoffs  between  simplicity,  thoroughness,  and  perceived  accuracy. 

In  the  succeeding  subsections,  we  will  present  the  proposed  methodologies,  emphasizing  areas  of 
subjectivity  and  listing  the  advantages  and  shortcomings  of  each.  There  are  valid  arguments  in  favor  of 
each  proposal.  Perhaps  the  final  cost/risk/benefit  methodology  will  evolve  into  some  hybrid  of  the 
proposed  methodologies. 

5.1  Decision  Tree  Methodology.  An  example  of  this  straightforward  approach  incorporates  a  series 
of  13  sample  questions  (see  Table  2)  pertaining  to  the  costs  and  benefits  of  full-up  LFT  of  a  system.  It 
represents  an  expert  system  of  sorts  in  that  the  answers  to  the  various  questions  will  lead  the  user  on  a 
path  that  will  eventually  provide  a  decision  concerning  whether  or  not  to  apply  for  a  waiver  from  full-up 
LFT,  while  simultaneously  suggesting  alternative  testing  strategies. 

The  initial  five  questions  listed  in  Table  2  pertain  to  cost.  We  felt  that  the  total  production  cost 
(determined  by  questions  1  and  2)  was  an  important  factor  in  the  decision  of  whether  or  not  to  conduct 
full-up  LFT,  since  one  would  be  less  inclined  to  perform  such  tests  on  an  expensive  system.  The  question 
as  to  what  is  meant  by  "expensive"  requires  an  answer  that  is,  of  course,  subjective.  Indeed,  subjectivity 
is  present  in  most  of  the  decision  rules;  and  we  will  discuss  this  in  greater  detail  later  in  this  section. 
Question  3  checks  on  the  availability  of  facilities  for  full-up  LFT,  and  questions  4  and  5  pertain  to  the  cost 
of  performing  such  tests. 
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Table  2.  Sample  Questions  Concerning  Costs/Benefits  of  Full-Up  LFT 


Costs 

1.  What  is  the  average  cost  of  a  system? 

2.  How  many  systems  are  to  be  produced? 

3.  What  is  the  cost  of  facilities  for  full-up  testing? 

4.  What  is  the  cost  of  conducting  a  noncatastrophic,  full-up  LFT? 

5.  What  is  the  probability  of  catastrophic  kill  given  a  full-up  LFT? 

Benefits 

6.  How  important  is  system  vulnerability  to  system  survivability? 

7.  What  percentage  of  the  developmental  cost  is  programmed  for  vulnerability  reduction  measures 
(for  example,  compartmentalized  ammunition)? 

8.  How  significant  are  the  synergistic  effects  of  damage  mechanisms  on  multiple  subsystems? 

9.  What  percentage  of  the  components  have  been  previously  tested  or  subjected  to  ballistic  threats 
in  combat? 

10.  What  percentage  of  the  subsystems  have  been  previously  tested  or  subjected  to  ballistic  threats 
in  combat? 

11.  Have  other  tests  or  analyses  been  conducted  which  enable  conclusions  to  be  drawn  concerning 
overall  system  vulnerability? 

12.  What  percentage  of  the  untested  components  are  projected  for  testing? 

13.  What  percentage  of  the  untested  subsystems  are  projected  for  testing? 


The  remaining  eight  questions  attempt  to  quantify  the  benefit  of  performing  full-up  LFT.  Question  6 
checks  to  see  if  the  system  is  likely  to  be  hit.  If  not,  then  perhaps  the  concern  over  the  Pk/h  is 
diminished.  Question  7  checks  on  vulnerability  reduction  measures.  The  response  to  this  question  should 
be  consistent  with  the  response  to  the  previous  question.  That  is,  if  the  system  is  not  likely  to  be  hit,  then 
little  money  should  have  been  programmed  for  vulnerability  reduction.  Question  8  covers  synergistic 
effects  of  damage  mechanisms  on  subsystems.  If  these  might  be  significant,  then  there  should  be  an 
inclination  towards  full-up  LFT.  Question  1 1  inquires  about  other  testing  and  analyses  (for  example, 
nondestructive  testing  and  computer  simulations).  Questions  9,  10,  12,  and  13  examine  previous  and 
projected  testing  on  components  and  subsystems.  Obviously,  if  knowledge  about  the  system  exists  from 
other  testing  and  analyses,  the  need  for  ftill-up  LFT  is  diminished. 

A  proposed  decision  tree  is  shown  in  Figure  6.  Notice  that  we  are  inclined  to  test  the  "inexpensive" 
systems  and  grant  a  waiver  for  the  "expensive"  systems.  In  the  middle,  where  many  systems  fall,  we  ask 
additional  questions.  A  magnification  of  the  boxed  area  in  Figiure  6  is  shown  in  Figure  7. 
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In  working  with  the  13  questions,  we  have  attempted  to  establish  reasonable  decision  rules.  For 
example,  question  1  relates  to  the  average  production  cost  of  a  system.  If  this  value  is  greater  than 
$100M,  we  are  inclined  to  waive  the  fiill-up  LFT  requirement;  whereas,  if  this  value  is  less  than  $50M, 
we  tend  to  perform  the  test.  However,  note  that  the  decision  concerning  full-up  LFT  is  not  based  solely 
on  the  answer  to  this  question.  Cost  values  between  $50M  and  $100M  require  additional  consideration 
and  some  systems  >$100M  are  live-fired  because  of  other  considerations.  These  threshold  values  were 
derived  by  examining  the  1989  Military  Cost  Handbook  published  by  Data  Search  Associates  of  Fountain 
Valley,  CA.  Data  from  this  handbook  are  shown  in  Figures  1 1, 12, 13,  and  14.  In  these  figures,  the  term 
"average  cost"  represents  the  costs  for  the  most  recent  (as  of  FY89)  specific  quantity  procurement  of  an 
item  excluding  spares.  Notice  that  support  vehicles  and  tracked  combat  vehicles  would  all  fall  into  the 
category  of  "less  than  $50M."  This  does  not  mean  that  they  would  all  be  subjected  to  full-up  LFT,  but 
it  does  indicate  a  likelihood  of  that  decision.  Many  of  the  aircraft  fall  into  the  same  category,  with  two 
(V22  Osprey  and  C17  CX)  being  "greater  than  $100M."  Another  half  dozen  require  more  detailed  study. 
Over  one-half  of  the  ships  fall  into  the  "greater  than  $100M"  category.  Based  on  these  data,  we  believe 
the  decision  rules,  incorporating  thresholds  of  $50M  and  $100M,  are  reasonable.  If  more  accurate  or 
recent  data  were  provided,  the  rules  should  be  reevaluated. 

Of  course,  the  decision  rules  are  the  most  subjective  feature  of  this  methodology.  They  should 
generate  much  thought  and  may  require  periodic  adjustments.  In  addition,  there  could  be  changes  to  the 
list  of  questions,  periiaps  rephrasing  them  in  order  to  replace  qualitative  decision  rules  with  quantitative 
ones.  However,  if  this  methodology  were  accepted  (after  any  desired  modifications),  it  would  offer  the 
following  advantages: 

•  It  is  appropriate  for  all  new  systems  without  modifications. 

•  It  can  be  implemented  at  MS  n  of  the  developmental  process. 

•  The  questions  are  straightforward  with  a  minimum  of  subjectivity. 

•  Questions  9-13  can  suggest  a  mix  of  testing  on  components  and  subsystems  necessary  to  justify 
a  waiver  from  full-up  LFT. 

•  It  provides  an  easy-to-understand  procedure  for  submission  of  a  waiver  from  fiill-up  LFT. 
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Figure  8.  Average  costs  of  tracked  combat  vehicles. 
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ure  9.  Average  costs  of  support  vehicles. 
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Figure  10.  Average  costs  of  aircraft. 
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Figure  11.  Average  costs  of  ships. 
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This  final  advantage  means  that  a  Project  Manager  who  wanted  to  request  a  waiver  from  full-up  LFT 
would  submit  the  following: 

(1)  A  formal  request  for  waiver. 

(2)  TTie  traU  through  the  decision  process. 

(3)  The  justification  for  decisions  at  each  node  of  the  decision  tree. 

(4)  The  threat  document  to  substantiate  such  decisions. 

(5)  Any  additional  backup  material  deemed  pertinent. 

We  emphasize  that  what  we  have  presented  is  an  example  of  the  decision  tree  methodology.  The 
actual  set  of  questions  might  be  modified,  and  the  rules  at  each  decision  node  could  be  altered.  We 
encourage  knowledgeable  input  in  both  cases.  However,  as  a  methodology,  the  decision  tree  remains  a 
feasible  procedure  for  incorporating  the  costs  and  benefits  of  conducting  full-up  LFT  of  a  system  in  order 
to  determine  whether  or  not  a  waiver  of  such  testing  is  justified.  Its  simplicity  is  appealing,  but  some  may 
feel  it  lacking  in  rigor.  A  reasonable  alternative  is  to  use  this  methodology  as  a  prefilter  to  examine 
systems  that  are  either  very  expensive  or  very  inexpensive.  For  those  systems  falling  between  these  two 
extremes,  a  different  more  thorough  methodology,  such  as  those  outlined  in  the  succeeding  sections,  might 
be  applied. 

5.2  Knowledge-Based  Methodology.  The  knowledge-based  methodology  is  based  upon  the  following 
principles: 

(1)  The  goal  of  analysis  is  to  provide  information  for  decision  makers.  For  example,  see  Operational 
Research  in  Practice:  Report  of  a  NATO  Conference  (Davis  and  Verhulst  1958).  The  author  is 
indebted  to  the  late  Art  Stein  for  this  reference. 

(2)  The  reason  for  testing  is  to  provide  data  that  is  needed  to  improve  analyses. 

(3)  There  are  many  data-gathering  techniques,  each  offering  benefits  in  quality  of  data  and  carrying 
its  own  price.  It  is  necessary  to  assemble  a  data-gathering  plan  that  uses  aU  expropriate 
techniques  to  obtain  data  of  sufficient  quality  at  a  cost  that  is  low  relative  to  other  plans. 
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To  implement  this  methodology,  it  is  necessary  to  show  that: 


•  Quality  of  data  can  be  quantified  and  related  to  testing. 

•  Decision  maldng  can  be  related  to  quality  of  vulnerability  information. 

•  Expected  costs  can  be  estimated. 

To  quantify  the  quality  of  data,  a  number  of  techniques  appear  to  be  useful.  (They  are  discussed  in 
section  5. 2. 1.2.)  In  this  section,  we  have  emphasized  one  in  which  data  are  represented  by  intervals  in 
which  the  data  provider  feels  the  true  value  lies.  These  intervals  are  amenable  to  quantitative 
interpretation  in  terms  of  accuracy  and  confidence. 

To  relate  decision  making  to  vulnerability  information,  the  methodology  concentrates  on  the  required 
vulnerability  information  set  (RVIS)  which  will  be  used  by  decision  makers.  Since  this  information  is 
normally  supplied  through  models,  the  relationship  between  data  quality  and  model  output  is  directly 
applicable  to  meeting  decision  needs. 

The  appropriate  cost  measure  is  an  expected  cost  that  reflects  the  probability  of  large  costs  accruing 
from  rmintended  outcomes  of  tests.  The  use  of  expected  costs  is  common  in  other  fields. 

A  hypothetical  example  ("The  Ballistic  Bicycle"),  which  demonstrates  several  features  of  the 
knowledge-based  methodology,  is  presented  in  section  5.2.3.  In  this  example,  three  methods  of 
quantifying  data  quality  are  used.  In  each  case,  the  quality  is  propagated  into  the  RVIS.  A  number  of 
test  plans  are  then  proposed  that  involve  different  mixes  of  component,  subsystem,  and  full-up  munition 
tests;  each  of  the  test  plans  is  refined  to  meet  the  RVIS  quality  requirements.  Given  they  meet  the  quality 
requirement,  through  analysis  of  expected  costs,  the  preferred  plan  becomes  obvious. 

A  second  hypothetical  example  (the  F-16)  is  based  upon  an  actual  analysis  performed  on  the  aircraft 
at  about  MS  n.  Although  differing  from  the  Ballistic  Bicycle  in  type  of  threat  analyzed  and  vulnerability 
analysis  technique  used,  the  F-16  example  employs  a  similar  data  quality  evaluation  technique. 
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It  is  important  to  note  that  the  knowledge-based  methodology  does  not  completely  eliminate 
subjectivity.  The  fact  that  this  analysis  must  be  made  early  in  the  development  cycle  of  a  system 
precludes  the  total  elimination  of  subjectivity.  However,  it  is  clear  that  this  methodology: 

•  Identifies  the  remaining  subjectivities. 

•  Supports  bounding  those  subjectivities  and  assessing  their  potential  impact. 

•  Directly  relates  the  subjective  elements  to  possible  testing. 

The  knowledge-based  methodology  offers  several  ancillary  benefits. 

•  It  requires  that  a  global,  coordinated  data-gathering  plan  be  assembled  and  analyzed  early  in  the 
development  cycle. 

•  It  requires  that  confidence  and  accuracy  in  both  data  and  analytical  results  be  evaluated. 

•  It  takes  advantage  of  and  augments  analyses  that  are  already  an  intrinsic  part  of  every  system 
development. 

5.2.1  Benefits  and  Expected  Cost  of  Data-Gathering.  Every  benefit/cost  methodology  obviously 
requires  definitions  and  quantification  schemes  for  the  benefit  being  sought  and  the  cost  of  the  various 
alternatives  for  achieving  that  benefit.  This  section  develops  quantification  schemes  for  the  benefits  and 
the  expected  cost  of  a  data-gathering  plan  including  testing.  First,  the  following  concepts  must  be  defined. 

5.2.1. 1  Data-Gathering  Plans.  In  this  section,  a  test  plan  or,  more  generally,  a  data-gathering  plan, 
refers  to  all  of  the  testing  and  other  information-gathering  activities  throughout  the  acquisition  cycle  of 
an  equipment  item.  A  list  of  potential  sources  of  information  includes: 

•  Prior  Data. 

•  Analysis. 

•  Component-Level  Tests. 
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Subscale  Testing. 


•  (Full-Scale)  Subsystem  Tests. 

•  Nondestructive  Tests. 

•  Full  System  Tests,  Without  Fuel/Ammo.* 

•  Full  System  Tests.* 

The  strategy  is  to  optimize  the  combination  of  these  data-gathering  methods  to  be  applied  to  any  given 
system. 

It  must  be  noted  that  the  concept  of  a  total  data-gathering  plan  is  applicable  at  any  point  in  the 
acquisition  cycle.  In  fact,  as  shown  in  a  preceding  section,  the  initial  TEMP  is  a  requirement  at  MS  I. 
Subsequent  to  MS  I,  critical  design  parameters  and  critical  technologies  are  defined  and  selected  for 
incorporation  into  the  concept  item.  Often,  this  process  involves  testing,  which  bears  upon  the  analysis 
of  the  vulnerability  of  the  emerging  system  and  which  reveals  critical  data  voids.  Thus,  the  ability  to 
foresee  the  need  for  specific  vulnerability  information  improves  throughout  the  development  cycle.  In 
particular,  at  MS  H,  it  is  generally  possible  to  formulate  a  data-gathering  plan  that  takes  into  account  all 
data  accumulated  to  date  and  anticipates  the  kinds  and  amounts  of  data-gathering  that  wHl  be  required. 
This  fits  well  in  the  sequence  of  analyses  contained  in  the  acquisition  cycle  (see  Figure  5). 

Note  that  it  is  not  necessary  to  have  a  highly  detailed  model  of  a  developmental  item  in  order  to 
anticipate  the  kinds  of  tests  that  will  be  required  to  meet  decision  data  needs,  hi  fact,  it  is  very 
appropriate,  at  MS  n,  to  use  coarse  and/or  surrogate  models  to  plan  for  tests  that  will  produce  data  to  be 
used  in  highly  detailed  analyses  at  MS  m.  Much  of  the  disagreement  that  has  arisen  within  the 
community  stems  from  confusion  regarding  this  point. 

5.2. 1.2  RVIS.  Tying  the  "benefit"  portion  of  the  methodology  to  the  quality  of  the  vulnerability  data 
that  will  be  used  in  decisions  raises  three  essential  questions: 


Includes  operational  tcsts/development  tests  (OT/DT). 
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(1)  What  are  the  vulnerability  data?  (Enumeration) 


(2)  How  is  its  "quality"  measured?  (Evaluation) 

(3)  What  "quality"  is  required?  (Requirement) 

In  fact,  it  is  the  answering  of  these  three  questions  that  constitutes  most  of  the  difficulty  in  the 
development  and  implementation  of  this  methodology.  Much  of  the  remainder  of  this  section  describes 
and  gives  an  example  of  a  few  methods  of  enumeration,  evaluation,  and  requirement.  For  this  subsection, 
it  is  enough  to  describe  the  underlying  concepts. 

We  refer  to  that  set  of  information  that  satisfies  the  requirement  for  vulnerability  information  of 
sufficient  quality  as  the  RVIS. 

Enumeration 

On  the  surface,  it  might  seem  that  the  potential  amotmt  of  vulnerability  data  is  infinite.  Indeed,  there 
are  an  infinite,  number  of  possible  combinations  of  fragment  masses  and  velocities,  blast  overpressure-vs.- 
time  curves,  and  other  target-threat  combinations.  More  subtle  are  the  hidden  vulnerabilities,  sometimes 
referred  to  as  the  "unknown  unknowns":  single  wires  and  unforeseen  leakage  flow  paths  and 
interconnected  subsystems  that  can  react  synergistically  to  produce  a  much  greater  response  to  a  threat 
than  expected.  These  clearly  cannot  be  enumerated. 

However,  if  looked  at  from  the  perspective  of  the  decision  maker,  there  is  a  finite  package  of 
vulnerability  information  that  will  be  used  in  studies  and  analyses  that  lead  to  a  decision.  As  depicted  in 
Figure  12,  that  information  is  combined  with  other  information  in  the  decision  process.  Often, 
intermediate  steps  are  involved:  for  example,  it  is  common  for  the  vulnerability  information  to  be 
combined  with  other  aspects  of  survivability  and  scenario  factors  in  simulations  and  war  games.  In  such 
cases,  enumeration  of  the  RVIS  is  rather  straightforward:  it  is  that  vulnerability  data  specified  in  the  input 
lists  of  the  simulations  and  war  games  being  used.  In  other  cases,  a  listing  of  the  required  vulnerability 
information  may  not  be  so  rigorous.  However,  under  any  circumstance,  it  is  clear  that  the  amount  of 
vulnerability  information  that  will  be  used  in  decision-making  processes  is  limited.  Thus,  concentration 
upon  satisfying  the  needs  of  decision  makers  makes  it  irrelevant  to  ask  questions  about  the  existence  of 
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Figure  12.  Roles  of  analysis  in  the  acquisition  cycle. 

unknown,  indeterminable  vulnerabilities:  if  decisions  will  be  made  without  their  inclusion,  it  is  consistent 
to  neglect  them  in  this  methodology. 

Vulnerability  data  requires  significant  processing  before  they  are  useful  for  decision  making:  a  single 
test  result  has  very  little  value  to  a  combat  model.  Rather,  if  combat  model  results  are  used  in  a  decision, 
the  vulnerability  data  will  have  to  be  in  a  form  such  as  single  shot  kUl  probabilities,  vulnerable  areas,  blast 
contours,  or  lethal  areas  for  the  attacking  munition.  These  data  must  come  from  vulnerability  analysts. 
This  observation  has  two  important  ramifications: 

(1)  The  source  of  quantified  lists  of  RVIS  is  the  supplying  vulnerability  analysts. 

(2)  Enumeration  of  the  elements  of  die  RVIS  is  a  matter  of  identifying  the  data  which  the  decision 
makers  require. 

Evaluation 

The  definition  and  quantification  of  quality  as  applied  to  the  RVIS  constitutes  the  major  research  in 
this  project.  Although  the  work  in  this  area  is  stiU  ongoing,  a  few  basic  guidelines  can  be  stated: 
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(1)  Quality  must  be  related  and  relatable  to  successful  decision  making.  This  affirms  the  heuristic 
notion  that  RVIS  quality  is  somehow  related  to  the  accuracy  and  importance  of  the  data. 

(2)  The  definition  and  the  quantification  of  quality  must  be  adaptable  to  all  kinds  of  military  systems 
and  threats. 

(3)  The  definition  and  the  quantification  of  quality  must  be  flexible  enough  to  take  advantage  of  the 
data  and  analyses  available  at  MS  H.  In  view  of  the  markedly  different  development  cycles 
pertinent  to  land,  sea,  and  air  platforms,  this  is  a  most  demanding  guideline. 

Generically,  the  approach  to  quantifying  quality  is  suggested  by  Figure  13.  The  concept  is  as  follows: 
Instead  of  inputting  specific  numbers  (point  estimates)  for  data  required  by  the  vulnerability  models,  the 
analyst  inputs  numbers  with  associated  indicators  of  their  quality.  The  model  propagates  the  quality  along 
with  each  datum.  The  output,  along  with  the  required  vulnerability  information,  is  an  expression  of  the 
quality  of  the  information.  For  example,  the  quality  of  the  data  to  be  used  by  the  analyst  may  be 
expressed  as  an  uncertainty  in  the  data  values.  The  ouq)uts  of  tiie  vulnerability  models,  which  constitute 
the  information  that  supports  acquisition  decisions,  will  have  an  associated  uncertainty  that  reflects  the 
quality  of  that  information.  To  be  justifiable,  testing  should  lead  to  a  lower  uncertainty  (higher  quality) 
in  the  output  information. 

In  practice,  the  currently  used  vulnerability  models  generally  do  not  provide  for  the  propagation  of 
uncertainty.  As  a  result,  the  analyst  may  have  to  propagate  uncertainty  by  other  means.  For  example, 
Monte  Carlo  techniques  in  which  specific  values  are  selected  from  a  range  of  values  for  each  input 
parameter  could  help  to  map  out  the  uncertainty  in  the  output.  It  may  also  be  possible  to  conduct 
simplified  vulnerability  calculations  in  which  the  uncertainty  is  mathematically  propagated. 

To  implement  any  system  of  quality  quantification,  it  is  necessary  to  adopt  a  mathematical  model  of 
quality/uncertainty.  Two  such  models  are  currently  under  consideration. 

One  of  the  more  promising,  rigorous  ways  to  define  quality  is  based  upon  the  statistical  concepts  of 
accuracy  and  confidence.  First,  based  upon  the  confidence  attributed  to  other  information  that  is  used  in 
the  supported  decision  processes,  we  choose  a  confidence  level.  (Commonly,  values  such  as  80%,  90%, 
95%,  and  in  extreme  cases  99%  are  used.)  ITien,  one  evaluates  a  vulnerability  datum  and  establishes  an 
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interval  within  which  that  datum  is  believed,  at  the  specified  level  of  confidence,  to  lie.  Such  an  interval 
is  called  a  confidence  interval;  clearly,  its  width  depends  not  only  upon  the  nature  of  the  Hatunn  being 
estimated,  but  on  the  associated  level  of  confidence.  For  example,  suppose  that  the  decisions  being 
supported  require  a  confidence  level  of  80%.  The  single  shot  kill  probability  (SSKP)  for  a  given  round 
against  the  subject  target  is  to  be  estimated.  In  addition,  an  interval  is  specified  around  that  SSKP  value 
such  that,  with  80%  confidence,  the  true  SSKP  is  believed  to  lie  within  the  interval.  The  width  of  the 
interval  reflects  the  accuracy  of  the  estimate;  the  confidence  (80%)  reflects  the  adequacy  of  the  interval. 


A  second  promising  approach  to  estimating  confidence  intervals  uses  the  mathematical  tools  popularly 
known  as  Fuzzy  Math"  or  'Fuzzy  Logic."  In  one  application  of  Fuzzy  Math,  the  exact  value  of  a 
parameter  is  acknowledged  as  unknown;  however,  the  actual  parameter  value  is  believed  to  lie  within  a 
specified  range.  As  shown  in  Figure  14,  a  larger,  encompassing  range  is  also  idaitified,  outside  of  which 
the  parameter  value  is  most  certainly  believed  not  to  lie.  These  ranges  form  a  trapezoidal  region, 
sometimes  called  a  "Membership  Function,"  that  describes  the  uncertainty  in  the  parameter’s  value.  Some 
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Figure  14.  A  typical  membership  function. 

current  effort  is  being  directed  toward  application  of  Fuzzy  Math  constructs  toward  the  uncertainty  in 
(gridcell  level)  vulnerability  data  and  the  use  of  Fuzzy  Math  tools  to  the  aggregation  of  that  data  into  an 
overall  quality  value  for  a  data  set  (Schlegal,  Shear,  and  Taylor  1985;  Taylor  and  Boswell  1989;  Celmins 
1993). 

As  discussed  in  Celmins  and  in  Schlegal  et  al.,  an  outstanding  benefit  of  the  Fuzzy  Mathematics 
approach  to  the  current  problem  is  in  the  ability  to  model  analyst  uncertainty,  a  quantity  that  most 
certainly  does  not  conform  to  the  rules  of  probability  and  common  statistics.  The  use  of  Fuzzy  Math 
precludes  the  use  of  many  of  the  powerful  tools  of  conventional  statistics.  However,  this  loss  is  more  than 
compensated  by  the  more  accurate  depiction  of  human  estimation  provided  by  a  membership  function.  In 
addition,  current  work  has  provided  mathematical  procedures  appropriate  to  the  manipulation  of  Fuzzy 
constracts. 

The  Ballistic  Bicycle  example  that  follows  originally  used  two  other  measures  of  quality.  First,  a 
heuristic  metric  was  proposed  in  which  the  importance  of  each  intermediate  datum  to  the  RVIS  is 
multiplied  by  an  evaluation  (point  estimate)  of  the  quality  of  that  datum  and  summed  to  give  an  overall 
confidence  in  the  RVIS.  Also,  for  the  pure  fiill-up  excursion,  a  parametric  estimate,  assuming  a  binomial 
distribution,  is  demonstrated. 
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These  examples  serve  to  show  that  the  measure  of  quality  need  not  be  fixed  in  order  to  apply  the 
benefit/cost  methodology  and  achieve  useful  results.  However,  progress  in  the  Fuzzy  Math  method  of 
quality  quantification  resulted  in  its  selection  for  most  of  the  Ballistic  Bicycle  example,  with  a  parametric 
(Binomial)  analysis  used  for  the  totally  experimental  test  plan.  Similarly,  the  F- 16  example  presents 
another  application  of  a  Fuzzy  Math  quantification. 

Requirement 

In  the  terms  developed  previously,  the  quality  requirement  can  be  specified  in  terms  of  two  numbers. 
First,  one  must  have  a  required-accuracy  value — a  number  that  expresses  the  bounds  within  which  one 
expects  to  find  the  true  value  of  an  estimated  datum.  Secondly,  one  must  have  a  certain  confidence  that 
the  specified  boimd  is  actually  large  enough.  Both  of  these  numbers  should  come  from  the  (most 
demanding)  supported  decision  processes.  In  practice,  the  confidence  value  is  set  a  priori  to  be  consistent 
with  confidence  expressed  in  the  other  inputs  to  the  decision.  The  accuracy  requirement  is  based  upon 
the  following  axiom: 

•  Since  decisions  depend  upon  many  inputs,  such  as  future  scenarios,  which  cannot  be  tested,  the 
vulnerability  data  need  not  be  "perfect."  However,  confidence  in  decisions  should  not  be  degraded 
by  (testable)  inaccinracies  in  the  vulnerability  mfoimation. 

Thus,  the  quality  requirement  is  totally  determined  by  the  supported  decision  processes,  as  it  should 
be.  Methods  for  obtaining  confidence  and  accuracy  numbers  include: 

•  Quantitative:  Sensitivity  Analyses  of  Decision  Models. 

•  Qualitative:  Judgment  of  Experienced  Decision  Analysts. 

5.2.1.3  Expected  Cost.  Defining  the  cost  of  a  data-gathering  plan,  especially  die  test  portion,  is 
difficult  because,  for  many  tests,  there  is  wide  latitude  in  the  possible  outcomes.  Furthermore,  there  may 
be  a  markedly  different  cost  associated  with  every  possible  outcome.  For  example,  an  xmexpected 
catastrophic  event  can  be  significantly  expensive,  a  particularly  important  point  for  U.S.  Navy  (USN)  and 
USAF  platforms. 
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However,  well-developed  techniques  exist  for  determining  an  expected  cost,  a  probability  weighted 
average  cost  that  accounts  for  the  unexpected.  For  example,  we  can  apply  some  rather  standard,  first-order 
concepts  as  follows: 

The  expected  cost  of  a  test  is  the  probability  of  a  given  outcome  times  the  cost  of  that  outcome, 
summed  over  all  outcomes.  The  expected  cost  of  data-gatheiing,  C,  is  the  expected  cost  of  one  test 
summed  over  all  tests. 


NTEST  OUTCOMES 

c-Y.  E  p,fi 

i.l  M 

P-j :  The  probability  of  outcome  j  in  test  i. 

Cj :  The  cost  to  the  program  of  outcome  j. 

5.2.2  TTie  Benefit/Cost  Procedure.  Given  the  concepts  listed  previously,  the  procedure  for  evaluating 
the  benefit/cost  ratio  of  an  information  gathering  plan  is  summarized  in  the  following  sections. 

5 .2.2. 1  Enumerate  the  RVIS .  List  the  vulnerability  information  being  supplied  to  the  most  demanding 
decision  process. 

5.22.2  Establish  the  Required  Quality  for  the  Decisions  to  be  Supported. 

(1)  Determine  the  confidence  that  is  commensurate  with  the  other  inputs  and  analyses  in  the  decision 
process. 

(2)  Determine  the  accuracy  necessary  in  the  vulnerability  information  to  assure  that  the  correctness 
of  the  decision  will  not  be  influenced  by  uncertainty  in  the  vulnerability  data. 

(3)  In  general,  select  the  most  demanding  accuracy  and  confidence. 
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5.2.23  Evaluate  the  Current  (Baseline)  Quality  of  the  RVIS. 


(1)  Adopt  one  or  more  procedures,  appropriate  for  the  various  data,  for  evaluating  data  quality. 

(2)  Evaluate  the  baseline  RVIS  quality. 

Given  that  the  baseline  is  insufficient: 

5.2.2.4  Construct  and  Evaluate  Data-Gathering  Plans. 

(1)  Analyze  the  baseline  to  determine  the  data  that  is  needed  to  improve  the  RVIS  quality. 

(2)  Develop  test  plans  to  acquire  needed  data. 

(3)  Evaluate  expected  RVIS  quality  with  data  anticipated  from  test  plans. 

(4)  Evaluate  expected  cost  of  test  plans. 

5.2.2.5  Compare  Cost-Effectiveness  of  Plans.  This  procedure  is  demonstrated  in  a  simplified  example 
("The  Ballistic  Bicycle")  in  section  5.2.3. 

5.2.3  The  Ballistic  Bicycle.  In  order  to  demonstrate  the  procedure  evolving  as  the  knowledge-based 
benefit/cost  methodology,  consider  a  simple,  h)q)othetical  case. 

The  "Ballistic  Bicycle"  (Figure  15),  being  considered  for  acquisition,  is  approaching  MS  n.  The 
bicycle  design  has  not  been  finalized.  However,  from  concept  designs  and  previous  bicycle  development 
programs,  a  number  of  features  are  roughly  known.  These  have  been  used  (and  refined)  since  MS  0  to 
conduct  analyses  that  have  guided  the  development,  with  increasing  detail  included  in  the  analyses  as  it 
has  become  available.  Now,  approaching  MS  n,  it  is  necessary  to  put  together  a  vulnerabUity-data- 
gathering  plan  to  assure  that  the  RVIS  needed  for  future  design  and  acquisition  decisions  will  be  available 
when  needed. 

The  subsequent  sections  correspond  to  the  procedure  elements  listed  in  section  5.2.2. 
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Figure  15.  The  Ballistic  Bicycle. 


5.2.3. 1  Enumerate  the  RVIS:  List  the  Vulnerability  Information  Being  Supplied  to  the  Most 
Demanding  Decision  Process. 

In  general,  the  scenario  in  which  the  Ballistic  Bicycle  will  be  used  is,  like  the  future  world  situation, 
most  uncertain.  However,  the  decision  to  buy  the  bike  will  hinge  largely  on  a  need  for  such  bicycles  in 
training  films  to  be  used  in  Third-World  coimtries.  It  was  noted  that  grenade  attacks  in  such  films 
invariably  involve  a  bicycle  innocently  propped  up  near  the  doorway  of  the  attacked  building.  Since  this 
scaiario  is  rather  well  developed,  we  take  this  as  the  most  demanding  in  terms  of  concomitant  accuracy 
and  confidence.  The  only  threat  weapon  which  will  enter  into  this  example  is  a  hand  grenade,  a  fragment 
and  blast  threat.  Typical  delivery  accuracy  for  the  grenade  is  in  the  order  of  a  6  m  CEP. 

Thus,  for  this  example,  the  data  required  is  the  single  roimd  kill  probability  (SRKP)  for  a  thrown 
grenade  (CEP  =  6  m)  against  the  Ballistic  Bicycle. 


5.2.3.2  Establish  the  Required  Quality  for  the  Decisions  to  be  Supported. 

(1)  Determine  the  confidence  that  is  commensurate  with  the  other  inputs  and  analyses  in  the  decision 
process. 


In  an  actual  application  of  this  methodology,  the  first  step  in  establishing  a  quality  requirement  for 
vulnerability  information  is  to  analyze  the  decision  processes  which  will  use  the  information.  Forced  to 
work  in  an  environment  of  gross  uncertainty  regarding  future  scenarios  and  other  decision  inputs,  decision 
makers  have  traditionally  not  given  much  formal  attention  to  the  degree  of  confidence  that  they  have  in 
their  data.  However,  it  is  reasonable  to  believe  that  analysis  of  past  decisions  and  assessment  of  decision 
models  can  lead  to  a  confidence  requirement  for  the  vulnerability  information.  For  this  example,  we  take 
80%  as  the  confidence  value  to  be  met. 

(2)  Determine  the  accuracy  necessary  in  the  vulnerability  information  to  assure  that  the  correctness 
of  the  decision  will  not  be  influenced  by  uncertainty  in  the  vulnerability 

Similarly,  although  decision  makers  have  traditionally  not  given  much  formal  attention  to  the  accuracy 
of  their  input  data,  analysis  of  past  decisions,  assessment  of  the  decision  models,  and  an  evaluation  of  the 
expected  accuracy  of  other  inputs  can  lead  to  an  accuracy  requirement  for  the  vulnerability  information. 
For  this  example,  we  take  .15  (i.e.,  ±7.5%)  as  the  accuracy  requirement. 

(3)  In  general,  select  the  most  demanding  accuracy  and  confidence. 

The  previous  factors  establish  the  RVIS  quality  requirement:  the  quality  must  be  equivalent  to  an  80% 

confidence  that  the  true  SRKP  for  the  grenade  against  the  Ballistic  Bicycle  probably  lies  within  an  interval 
of  .15. 

5.2.3.3  Evaluate  the  Current  (Baseline)  Quality  of  the  RVIS. 

(1)  Adopt  one  or  more  procedures,  appropriate  for  the  various  data,  for  evaluating  data  quality. 

In  evaluating  the  quality  of  the  RVIS,  it  appears  that  one  can  take  either  the  confidence  or  the 
accuracy  as  the  fixed  parameter  and  evaluate  the  other. 
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For  the  Fuzzy  Math  analysis,  the  collected  data  are  of  the  form:  "Give  me  the  range  within  which 
you  think  the  true  value  of  the  datum  "probably"  lies.  Give  me  the  extremes  outside  of  which  the  datum 
"surely"  doesn’t  lie."  (The  quantification  of  "probably"  and  "surely"  as  50%  and  99%  confidence  bounds 
and  the  calibration  of  respondents  are  discussed  in  Appendix  A.)  Thus,  we  are  fixing  the  confidence  level 
and  determining  how  large  the  corresponding  intervals  must  be  in  order  to  claim  an  80%  confidence  that 
the  true  SRKP  is  contained  therein.  (The  interpolation  to  an  80%  confidence  is  also  presented  in 
Appendix  A.) 

(2)  Evaluate  the  Baseline  RVIS  Quality. 

In  the  Baseline  case  (state  of  knowledge  at  MS  11),  the  best  means  of  evaluating  the  SRKP  for  the 
Ballistic  Bicycle  is  via  an  analytical  technique  such  as  FULL  SPRAY  in  which  independent  blast  and 
fragment  vulnerabilities  are  combined  to  produce  a  combined  blast-fragment  kill  assessment. 

For  the  Fuzzy  Math  exercise,  a  simplified  Full-Spray-like  analysis  was  performed.  A  simplified, 
FORTRAN  code  was  written  to  combine  the  data  and  to  propagate  the  uncertainties;  the  methodology  and 
code  are  described  in  Appendix  B.  (Simplifications  include  ignoring  air  drag  on  the  (single  mass) 
fragments.  Barrier  penetration  was  not  considered  important  for  a  bicycle.)  The  relative  importance  of 
the  blast  and  Iragment  kill  mechanisms  need  not  be  assumed;  they  are  intrinsically  taken  into  account  by 
the  (Survivor-Rule)  procedure  in  the  code,  which  assumes  independent  effects.  Blast-fragment  synergism 
was  not  addressed. 

Evaluation  of  the  Baseline  RVIS  quality  begins  with  an  analysis  of  the  state  of  knowledge  of  fragment 
and  blast  vulnerabilities  at  MS  H. 

Recall:  The  issue  for  the  knowledge-based  benefit/cost  evaluation  methodology  is  not  whether 
a  datum  is  known  at  MS  H.  Rather,  the  issue  is,  How  well  will  a  datum  be  known  when  it  will 
be  needed?  TTius,  identification  of  the  precise  components  that  will  make  up  the  final  Ballistic 
Bicycle  may  not  be  relevant.  More  relevant  is.  Will  good  estimates  of  the  vulnerability  of  fliose 
components  be  possible  when  they  are  identified? 


*  Computer  Program  for  General  Full  Spray  Materiel  MAE  Computations,  Volume  I— User  Manual,  JMEM  Report,  6 1 JTCG/ME- 
79-1-1. 
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The  following  text  will  extensively  reference  Figures  16  and  17.  The  former  portrays  a  grid  on  the 
Ballistic  Bicycle  which  will  be  used  for  enumeration  and  calculation  of  fragment  data.  Blast  data 
(assumed  azimuthally  isotropic)  will  be  enumerated  by  miss  distance,  as  portrayed  in  the  latter. 

The  status  of  pertinent  data  at  MS  II  is  shown  in  Table  3.  In  the  table,  pertinent  data  is  broken  into 
confidence  in  component  values  and  confidence  in  ability  to  predict  synergistic  effects.  In  the  latter 
portion  of  the  table,  the  number-letter  combinations  refer  to  Figure  16. 

In  each  case,  the  data  is  expressed  as  a  (qualitative)  point  estimate  of  confidence  and  as  a  range  of 
(Fuzzy  Math)  values  for  fragment  kills.  The  Fuzzy  Math  values  (a,  a,  b,  and  p)  refer  to  single  fragment 
Pk/h. 

The  (hypothetical)  rationale  for  the  numbers  in  Table  3  is  as  follows.  In  general,  there  is  good 
capability  to  predict  the  effects  of  fragments  on  tubular  components.  A  need  for  data  is  anticipated  for 
specialized  components,  such  as  brake  calipers,  which  have  not  yet  been  finalized  at  MS  n.  The  ability 
to  predict  synergistic  effects  is  worse  than  the  ability  to  predict  component  damage.  (An  example  of  a 
sjTiergistic  effect  occurs  in  grid  cell  4C  in  which  a  fragment  that  strikes  the  brake  caliper  could  leave  the 
caliper  undamaged,  yet  rotate  the  caliper  into  the  spokes  such  that  the  bicycle  is  left  immobile.) 


ABCDEFGH  I  J 


Figure  16.  The  Ballistic  Bicycle:  fragment  grid. 
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Table  3.  Status  of  Data:  MS  II 


Component  Pj^ 

Component 

Confidence 

a 

a 

b 

P 

Tubular 

Good 

.05 

.8 

.9 

.05 

Brake  Levers 

Good 

.05 

.45 

.5 

.05 

Brake  Cables 

Fair 

.1 

.7 

.9 

.1 

Brake  Calipers 

Poor 

.2 

.3 

.7 

.1 

Seat 

Poor 

.0 

.0 

.0 

.0 

Spokes 

Fair 

.1 

.2 

.5 

.1 

Sprocket 

Poor 

.2 

.2 

.7 

.1 

Chain 

Poor 

.2 

.3 

.8 

.2 

Wheel 

Poor 

.1 

.4 

.9 

.1 

Synergistic 

Grid  Sq. 

Confidence 

a 

a 

b 

P 

IE 

.0 

.1 

.1 

3C 

.2 

.2 

.5 

.2 

3H 

.2 

.2 

.5 

.2 

4C 

.2 

.2 

.6 

.2 

4E 

.1 

.1 

.4 

.3 

4F 

Good 

.0 

.0 

.2 

.2 

4H 

Poor 

.1 

.2 

.6 

.4 

5H 

Poor 

.1 

.2 

.5 

.3 

Blast  P^ 

Miss  Dist  (m) 

Confidence 

a 

a 

b 

P 

3 

Good 

0. 

1. 

1. 

0. 

4 

Good 

5 

Fair 

0.80 

6 

Poor 

0.65 

7 

Poor 

0.2 

0.50 

0.3 

8 

Poor 

0.15 

0.15 

0.30 

0.3 

9 

Fair 

0.1 

0.15 

0.2 

Fair 

0.05 

0.05 

0.1 

0.1 

11 

Good 

0.02 

0.02 

0.05 

0.05 

12 

Good 

0. 

0. 

0.01 

0.02 

13 

Good 

0. 

0. 

0. 

0. 

14 

Good 

0. 

0. 

0. 

0. 
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Using  these  values  and  the  code  described  in  Appendix  B,  an  estimate  of  the  quality  of  the  current 
(baseline)  vulnerability  information  was  obtained.  In  that  analysis,  the  Fuzzy  SRKP  was  found  to  be: 


SRKP  =  0.59,  0.76,  0.89,  0.92. 


In  accordance  with  the  gauge  established  in  Appendix  A,  this  corresponds  to  an  80%  confidence 
interval  of  0.21.  Since  this  corresponds  to  an  imcertainty  of  about  25%,*  we  conclude  that  a 
data-gathering  plan  is  needed  to  meet  the  requirement  of  the  RVIS. 


It  was  also  of  interest  to  investigate  the  relative  contributions  of  the  blast  and  fragment  contributions 
to  the  result.  The  structure  of  the  code  allowed  easy  substitution  of  zeroes  for  blast  and  fragment 
vulnerability.  Using  this  technique  gave: 


SRKP 

a  -  a 

a 

b 

b  +  ^ 

"80%  int." 

"%  unc."* 

Total 

MM 

■■ 

25 

Blast  Only 

mSsM 

51 

Frags  Only 

0.54 

0.72 

0.88 

28 

From  the  above,  we  note  that  blast  alone  is  about  two-thirds  as  effective  as  fragments  alone. 

5.2.3.4  Construct  and  Evaluate  Data-Gathering  Plans. 

(1)  Analyze  the  baseline  to  determine  the  data  that  is  needed  to  improve  the  RVIS  quality. 

Analysis  of  Table  3  shows  that  the  major  causes  of  uncertainty  in  the  results  are  from  the  fragments. 
Thus,  test  plans  that  are  aimed  to  improve  the  quality  of  the  data  must  anticipate  addressing  shortcomings 
in  fragment  data,  particularly  on  brake  components  and  on  synergistically  sensitive  regions  where  the 
brake  and  motive  components  are  in  close  proximity  with  wheel  components.  Blast  data  is  lacking  for 
the  bicycle  as  a  whole  (the  ability  to  evaluate  "sail  area"  and  translational  response  is  lacking). 


Percentages  calculated  with  respect  to  the  SRKP  value  that  results  from  using  the  center  values  of  all  Fuzzy  inputs.  This  value 
differs  slightly  from  the  center  value  of  the  Fuzzy  ouqjut.  See  Appendix  C. 
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Again,  note  the  context  of  the  foregoing  analysis.  Here  at  MS  n,  the  actual  vulnerability  data  is  not 
expected  to  be  known  since  many  of  the  components  have  not  been  finalized.  Rather,  the  discussion 
concerns  the  ability  to  know  that  data  when  it  will  be  needed  for  future  decisions. 

(2)  Develop  test  plans  to  acquire  needed  data. 

For  this  example,  three  different  data-gathering  plans  were  formulated  that  could  alleviate  the 
shortcoming  in  vulnerability  data. 

Plan  1 :  Plan  1  involves  tests  of  fragments  against  components,  firagments  against  subsystems,  and 
blast  against  the  whole  bicycle.  No  "full-up"  tests  (actual  grenade  against  a  full-up  bicycle)  are  included. 
The  purpose  of  Plan  1  is  to  improve  the  data  listed  in  Table  3. 

Plan  2:  Plan  2  involves  tests  of  fragments  against  components.  However,  instead  of  separate 
subsystem  (fragment)  and  blast  tests,  some  actual  grenade  tests  will  be  used.  Like  Plan  1,  the  purpose 
of  Plan  2  is  to  improve  the  data  listed  in  Table  3. 

Plan  3:  Plan  3  embodies  an  entirely  different  approach.  In  it,  only  actual  grenade  tests  are  used.  The 
number  of  tests  will  be  chosen  in  order  to  achieve  the  required  confidence  in  the  (directly  measured) 
SRKP. 

The  tests  to  be  conducted  are  listed  in  Tables  4-6. 

Postulate:  In  Test  Plan  1  shots  involving  blast  alone,  at  the  chosen  distances  (removed  from  the 
sure-kill  blast  region),  the  bicycle  can  be  protected  from  permanent  damage  without  loss  of  kill  data. 
Probability  of  loss  of  bicycle  =  0.10. 

Postulate:  Sufficient  fragments  from  the  grenade  tests  will  impact  the  bicycle  to  allow  assessment  of 
synergistic  effects.  It  is  believed  that  the  number  of  possible  exposures  (12  shots)  will  compensate  for 
the  lack  of  precisely  controlled  test  conditions. 
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Shots  in  Test  Plans  1  and  2  are  replicated  three  times. 


Note:  As  discussed  in  section  3,  the  shot  selection  in  Test  Plan  3  is  intended  to  concentrate  on  the 
areas  of  most  uncertainty.  Since  both  fragment  and  blast  kills  will  be  intrinsically  combined,  this  area  of 
uncertainty  is  expected  to  be  displaced  outward  from  the  regions  of  maximum  uncertainty  of  blast  or 
fragments  alone. 

(3)  Evaluate  expected  RVIS  quality  with  data  anticipated  from  test  plans. 

In  this  section,  we  evaluate  the  expected  quality  of  vulnerability  information  that  would  be  available, 
when  needed,  should  each  of  the  test  plans  be  adopted. 

Test  Plan  1.  As  a  result  of  Test  Plan  1,  the  expected  improvement  in  data  and  the  associated  change 
in  interval  widths  is  shown  in  Table  7. 

With  these  Fuzzy  values  input  to  the  (Appendix  B)  computer  code,  the  Fuzzy  SRKP  to  be  anticipated, 
if  Test  Plan  1  were  to  be  adopted,  is  found  to  be: 

SRKP  =  0.78,  0.82,  0.87,  0.89. 

In  accordance  with  the  gauge  established  in  Appendix  A,  this  corresponds  to  an  80%  confidence 
interval  of  0.08.  Since  this  corresponds  to  an  uncertainty  of  about  9%,  we  conclude  that  Test  Plan  1  will 
provide  adequate  improvement  to  meet  the  requirement  of  the  RVIS. 

We  again  recall  that  the  absolute  values  of  the  SRKP  are  relatively  meaningless:  At  MS  11,  many  of 
the  components  haven’t  even  been  designed;  thus,  the  center  values  (the  midpoints  of  the  inner,  "probable" 
ranges)  of  many  of  the  inputs  were  rather  subjective  estimates.  However,  the  estimates  of  future 
imcertainty,  as  expressed  in  the  widths  of  the  "probable"  and  "surely"  intervals,  are  meaningful  measures. 
Hence,  the  use  of  the  width  of  the  SRKP  Fuzzy  intervals  are  useful  indicators  of  expected  RVIS  quality. 

One  should  note  that  the  SRKP  center  value  anticipated  after  Test  Plan  1  differs  slightly  from  that  of 
the  baseline  even  though  the  center  values  of  all  the  inputs  were  kept  unchanged.  This,  of  course,  reflects 
the  mherent  nonlinearity  in  a  vulnerability  code,  even  one  as  simple  as  that  of  Appendix  B.  It  can 
similarly  be  shown  that  the  widths  of  Fuzzy  ou^ut  intervals  can  depend,  albeit  insensitively,  upon  the 
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Table  7.  Anticipated  Status  of  Data:  Post  Test  Plan  1 


Component  Pjr 

Component 

Confidence 

a 

a 

b 

P 

Tubular 

Good 

.05 

.8 

.9 

Brake  Levers 

Good 

.05 

.45 

.55 

Brake  Cables 

Fair 

.1 

.7 

.9 

.1 

Brake  Calipers 

Good 

.05 

.45 

.55 

.05 

Seat 

Poor 

.0 

.0 

.0 

.0 

Spokes 

Fair 

.1 

.2 

.5 

.1 

Sprocket 

Good 

.1 

.4 

.5 

.1 

Chain 

Good 

.1 

.5 

.6 

.1 

Wheel 

Good 

.1 

.6 

.7 

.1 

Synergistic  Pjr 

Grid  Sq. 

Confidence 

a 

a 

b 

P 

IE 

Good 

.0 

.1 

.1 

3C 

Good 

.1 

.3 

.4 

.1 

3H 

Good 

.1 

.3 

.4 

.1 

4C 

Fair 

.1 

.3 

.5 

.1 

4E 

Good 

.1 

.2 

.3 

.1 

4F 

Good 

.2 

.2 

4H 

Fair 

.1 

.3 

.5 

.1 

5H 

Fair 

.1 

.25 

.45 

.1 

Blast  Pjr 

Miss  Dist  (m) 

— 

Confidence 

a 

a 

b 

P 

3 

Good 

0. 

1. 

1. 

0. 

4 

Good 

0.05 

0.80 

0.95 

0.05 

5 

Good 

0.65 

0.75 

0.05 

6 

Good/Fair 

0.1 

0.50 

0.65 

0.1 

7 

Good 

0.05 

0.30 

0.40 

0.05 

8 

Good/Fair 

0.1 

0.15 

0.30 

0.1 

9 

Good 

0.05 

0.10 

0.15 

0.05 

10 

Good/Fair 

0.05 

0.05 

0.1 

0.05 

11 

Good 

0.02 

0.02 

0.05 

0.05 

12 

Good 

0. 

0. 

0.01 

0.02 

13 

Good 

0. 

0. 

0. 

0. 

14 

Good 

0. 

0. 

0. 

0. 

center  values  of  the  input  data.  Thus,  since  the  center  values  may  be  subjective  estimates,  it  would  be 
incorrect  to  emphasize  the  precise,  absolute  value  of  the  output  imcertainty  interval.  However,  it  can  also 
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be  shown  that  the  relationship  between  the  input  center  values  and  the  output  intervals  is  sufficiently 
insensitive  to  justify  the  loose  quantitative  comparisons  that  are  needed  for  the  benefit/cost  application 
developed  here. 

Test  Plan  2.  The  improvement  over  the  baseline  to  be  anticipated  from  Test  Plan  2  is  shown  in 
Table  8.  Note  that  the  use  of  full  grenades  in  the  blast  tests  in  lieu  of  subsystem  tests  results  in  more 
anticipated  data  on  fragments  against  components.  However,  the  anticipated  improvement  in  synergistic 
data  was  less.  In  addition,  the  difficulties  involved  in  the  blast  measurements  and  the  possible  loss  of  blast 
data  due  to  (independent)  fragment  effects  resulted  in  a  lower  confidence  (larger  Fuzzy  intervals)  in  some 
of  the  blast  data. 

With  these  Fuzzy  values  input  to  the  (Appendix  B)  computer  code,  the  Fuzzy  SRKP  to  be  anticipated 
if  Test  Plan  2  were  to  be  adopted  is  found  to  be: 

SRKP  =  0.79,  0.83,  0.87,  0.89. 

In  accordance  with  the  gauge  established  in  Appendix  A,  this  corresponds  to  an  80%  confidence 
interval  of  0.07.  Since  this  corresponds  to  an  uncertainty  of  about  8%,  we  conclude  that  Test  Plan  2  will 
provide  adequate  improvement  to  meet  the  requirement  of  the  RVIS. 

Test  Plan  3.  As  a  result  of  Test  Plan  3,  the  expected  improvement  in  the  SRKP  is  found  to  be 


SRKP  =  0.85  ±  0.06. 


This  result  is  based  upon  anticipated  (binary)  results  and  a  binomial  error  analysis.  The  calculation 
is  presented  in  Appendix  D.  As  with  Test  Plans  1  and  2,  Test  Plan  3  satisfies  the  criterion  for  adequate 
improvement. 

(4)  Evaluate  expected  cost  of  test  plans. 

In  order  to  evaluate  the  expected  cost  of  the  three  test  plans,  some  basic  costs  will  have  to  be 
hypothesized.  These  are  contained  in  Table  9. 
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Table  8.  Anticipated  Status  of  Data:  Post  Test  Plan  2 


Component 

Component 

Confidence 

a 

a 

b 

P 

Good 

.8 

.9 

Brake  Levers 

Good 

.45 

.55 

Brake  Cables 

Fair 

.1 

.7 

.9 

.1 

Brake  Calipers 

Good 

.45 

.55 

.05 

Seat 

Fair 

.0 

Spokes 

Good 

.3 

.4 

.05 

Sprocket 

Good 

.1 

.4 

.5 

.1 

Chain 

Good 

.1 

.5 

.6 

.1 

Wheel 

Good 

.1 

.6 

.7 

.1 

Synergistic  P^ 

Grid  Sq. 

Confidence 

a 

a 

b 

P 

IE 

Good 

.0 

.0 

.1 

.1 

3C 

Good 

.1 

.3 

.4 

.1 

3H 

Good 

.1 

.3 

.4 

.1 

4C 

Fair 

.1 

.3 

.5 

.1 

4E 

Good 

.1 

.15 

.35 

.2 

4F 

Good 

.0 

.0 

.2 

.2 

4H 

Fair 

.1 

.3 

.5 

.1 

5H 

Fair 

.1 

.25 

.45 

.1 

Blast  Pjf 

Miss  Dist  (m) 

Confidence 

a 

a 

b 

P 

3 

Good 

0. 

1. 

1. 

0. 

4 

Good 

0.05 

0.80 

0.95 

0.05 

5 

Good/Fair 

0.1 

0.60 

0.80 

0.1 

6 

Good/Fair 

0.1 

0.50 

0.65 

0.1 

7 

Good/Fair 

0.1 

0.30 

0.40 

0.1 

8 

Good/Fair 

0.1 

0.15 

0.30 

0.1 

9 

Good/Fair 

0.1 

0.10 

0.15 

0.1 

10 

Good/Fair 

0.05 

0.05 

0.1 

0.1 

11 

Good/Fair 

0.02 

0.02 

0.05 

0.1 

12 

Good 

0. 

0. 

0.01 

0.02 

13 

Good 

0. 

0. 

0. 

0. 

14 

Good 

0. 

0. 

0. 

0. 
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Table  9.  Basic  Costs  Associated  With  Testing 


Testing  Costs 

(including  Mimition,  Data  Analysis, .,.) 

Fragment  Test  (Setup) 

$1,000 

Fragment  Test  (1  Replication) 

$100 

Blast  Test  (Setup) 

$1,500 

Blast  Test  (1  Replication) 

$100 

Full-Up  Grenade  Test  (Setup) 

$2,500 

Full-Up  Grenade  Test  (1  Replication) 

$1,000 

Cost  of  Target 

Bicycle  Frame 

$400 

Wheels,  each 

$100 

Brakes 

$100 

Chain 

$100 

Drive 

$200 

Bicycle,  Assembled 

$1,000 

It  is  also  necessary  to  estimate  the  probability  of  loss  of  a  target  part  in  a  given  test.  Consistent  with 
the  assumptions  made  previously,  we  assume  that  grenade  tests  (all  of  which  are  against  the  full  bicycle) 
will  have  a  50%  probability  of  total  loss  of  the  target.  All  component/subsystem  tests  have  a  90%  chance 
of  loss  of  target  item. 

Additional  probabilities  are  required  for  fee  possible  synergistic  damage  regions,  such  as  grid  cell  4C 
in  which  various  amounts  of  damage  could  occur.  For  this  simple  example,  we  will  limit  fee  possible 
damage  sets  to: 


GRID  CELL  4C  (3  SHOTS) 

Only  brake  damaged,  every  shot  0.6 

Only  frame  damaged,  every  shot:  0.3 

GRID  CELL  4H  (6  SHOTS) 

Only  brake  damaged,  every  shot:  0.6 

Frame,  wheel,  and  chain  damage:  0.3 


In  Tables  10,  11,  and  12,  target  costs  are  presented  in  the  form  "cost  (probability)." 

Test  Plan  I.  Test  and  target  costs  reflect  three  replications. 

Test  Plan  2.  Test  and  target  costs  reflect  three  replications. 

Test  Plan  3.  Test  Plan  3  consists  only  of  a  series  of  full  grenades. 

5.2.3.5  Compare  Cost-Effectiveness  of  Plans.  The  LFT  knowledge-based  benefit/cost  evaluation 

methodology  specifies  that  the  data-gathering/test  plans  only  include  those  that  do  satisfy  the  RVIS  quality 
requirement.  In  this  example,  all  three  test  plans  are  acceptably  effective.  On  the  other  hand,  there  is  a 
significant  cost  difference  in  the  three  plans.  The  expected  costs  of  Plans  2  and  3  are  60%  and  180% 

more  expensive  than  Test  Plan  1,  respectively.  The  additional  costs  stem  from  the  use  of  full-up  grenade 

shots.  These  costs  accrue  in  two  ways.  The  tests  themselves  are  significantly  more  expensive  than 
component/subsystem  tests,  since  data  requirements  are  equal  to  the  sum  of  fragment  and  blast  tests,  and 
safety  requirements  make  each  replication  quite  expensive.  In  addition,  the  probability  (0.5)  of  losing  the 
entire  bicycle  on  a  shot  adds  significantly  to  the  overall  expected  cost.  In  essence,  the  fuU-up  grenade 
shots  carry  too  great  a  probability  of  frame  damage  due  to  fragments.  This  damage  is  expensive,  but 
yields  data  that  is  extraneous  due  to  previous  knowledge. 

5. 2.3.6  Discussion  of  Ballistic  Bicycle  Results.  The  preceding  section  concludes  that  fuU-up  grenade 
shots  added  significantly  to  the  costs  of  the  test  plans,  pointing  out  that  hypothesized  previous  knowledge 
negated  the  value  of  much  of  the  data  gained.  That  is,  given  a  different  starting  knowledge  base,  the 
conclusions  might  have  been  different. 

It  should  also  be  pointed  out  that  the  Ballistic  Bicycle  is  a  very  simple  target.  It  has  no  fuel  or 
ammunition,  the  impact  of  which  could  begin  a  chain  of  reactions.  In  addition,  it  is  "flat:"  components 
do  not  screen  each  other.  This  flatness  also  reduces  the  potential  for  complex  synergistic  interactions: 
components  in  a  two-dimensional  array  have  fewer  close  neighbors  than  do  those  in  three  dimensions. 
This  simplicity  translates  into  enhanced  confidence  in  a  model-based  analysis  of  the  vulnerability  of  the 
Ballistic  Bicycle:  the  possible  existence  of  "unknown  unknowns"  is  not  a  factor  in  the  confidence  ascribed 
to  the  vulnerability  analysis. 
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Table  10.  Test  Plan  1 


Table  11.  Test  Plan  2 


Table  12.  Test  Plan  3 
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The  conclusion  to  be  drawn  from  the  previous  observations  is  that  each  new  item  of  equipment 
introduces  its  own  relationship  between  a  priori  knowledge,  item  complexity,  and  experimental  costs.  It 
is  not  possible  to  make  a  categorical  statement  about  the  benefit-vs.-cost  of  any  class  of  testing  that  will 

pertain  to  every  potential  item. 

5.2.4  Summary  of  the  Knowledge-Based  Benefit/Cost  Evaluation  Methodology.  In  summary,  the 
benefit/cost  evaluation  methodology  is  based  upon  die  following  principles: 

(1)  The  goal  of  analysis  is  to  provide  information  for  decision  makers. 

(2)  The  reason  for  testing  is  to  provide  data  that  are  needed  to  improve  analyses.  Since  there  are 
many  data-gathering  techniques,  each  offering  benefits  in  quality  of  data  and  carrying  its  own 
price,  it  is  necessary  to  assemble  a  data-gathering  plan  that  uses  whichever  techniques  are 
beneficial  to  obtain  data  of  sufficient  quality  at  a  cost  that  is  low  relative  to  other  plans. 

To  implement  this  methodology,  it  is  necessary  to  show  that: 


•  Quality  of  data  can  be  quantified  and  related  to  testing. 

•  Decision  making  can  be  related  to  quality  of  vulnerability  information. 

•  Expected  costs  can  be  estimated. 

To  quantify  the  quality  of  data,  a  number  of  techniques  appear  to  be  useful.  In  this  report,  we  have 
emphasized  one  in  which  data  is  represented  by  intervals  in  which  the  data  provider  feels  the  true  value 
lies.  These  intervals  are  amenable  to  quantitative  interpretation  in  terms  of  accuracy  and  confidence. 

To  relate  decision  making  to  vulnerability  information,  the  methodology  concentrates  on  the  RVIS  that 
will  be  used  by  decision  makers.  Since  this  is  normally  supplied  through  models,  the  relationship  between 
data  quality  and  model  output  is  directly  applicable  to  meeting  decision  needs. 

The  appropriate  cost  measure  is  an  expected  cost  that  reflects  the  probability  of  large  costs  accruing 
from  unintended  outcomes  of  tests.  The  use  of  expected  costs  is  common  in  other  fields. 
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The  (hypothetical)  "Ballistic  Bicycle"  demonstrates  several  features  of  the  beaiefit/cost  evaluation 
methodology.  In  the  example,  three  methods  of  quantifying  data  quality  were  used.  In  each  case,  the 
quality  was  propagated  into  the  RVIS.  A  number  of  test  plans  were  then  proposed,  which  involved 
different  mixes  of  component,  subsystem,  and  full-up  mimition  tests;  each  of  the  test  plans  was  refined 
to  meet  the  RVIS  quality  requirements.  Finally,  through  analysis  of  expected  costs,  the  preferred  plan 
became  obvious.  In  addition,  analysis  of  the  example  showed  that  the  conclusion  was  dependent  upon 
the  relationship  between  a  priori  knowledge,  item  complexity,  and  experimental  costs. 

It  is  important  to  note  that  the  benefit/cost  evaluation  methodology  does  not  completely  eliminate 
subjectivity.  The  fact  that  this  analysis  must  be  made  early  in  the  development  cycle  of  a  system 
precludes  the  total  elimination  of  subjectivity.  However,  it  is  clear  that  this  methodology: 

•  Identities  the  remaining  subjectivities. 

•  Supports  bounding  those  subjectivities  and  assessing  their  potential  impact. 

•  Direcdy  relates  the  subjective  elements  to  possible  testing. 

The  benetit/cost  evaluation  methodology  offers  several  ancillary  benefits. 

•  It  requires  that  a  global,  coordinated  data-gathering  plan  be  assembled  and  analyzed  early  in  the 
development  cycle. 

•  It  requires  that  confidence  and  accuracy  in  both  data  and  analytical  results  be  evaluated. 

•  It  takes  advantage  and  augments  analyses  that  are  already  an  intrinsic  part  of  every  system 
development. 
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5.3  Knowledge-Based  Application  to  the  23  mm  vs.  F-16  (1981). 


5.3.1  Introduction. 

Scope 

The  Ballistic  Bicycle  example  provides  an  understanding  of  the  knowledge-based  benefit/cost 
methodology— particularly  an  appreciation  of  the  flexibility  in  choices  of  vulnerability  methodologies  and 
quality  quantification  techniques.  However,  it  remains  to  be  shown  that  the  technique  can  be  applied  to 
an  actual  case.  For  this  reason,  an  application  of  the  knowledge-based  benefit/cost  methodology  was  made 
to  the  F-16  Fighter  Aircraft — in  particular,  to  the  vulnerability  of  the  F-16  to  direct  hits  by  23-mm  high- 
explosive  incendiary  (HEI)  rounds. 

In  order  to  keep  this  overall  report  unclassified  and  yet  expose  all  assumptions  and  data,  some  of 
which  are  classified,  it  was  decided  to  augment  this  report  with  a  classifled  supplement.  This  section  will 
therefore  contain  only  unclassified  discussions  of  assumptions,  procedures,  and  relative  results  such  as 
percent  reductions  in  confidence  intervals. 

IT  MUST  BE  NOTED  THAT  THIS  STUDY  IS  NOT  AN  ACTUAL  LFT  WAIVER  ANALYSIS. 

The  purpose  of  this  exercise  is  to  scope  the  level  of  difficulty  to  be  encountered  in  the  application 
of  the  knowledge-based  benefit/cost  methodology  to  an  actual  system.  The  values  used  were 
derived  from  sources  as  identified  in  the  classified  supplement  to  this  report.  In  an  actual  LFT 
waiver  analysis,  such  resources  as  are  available  to  the  system  project  manager — ^who  is  ultimately 
responsible  for  any  waiver  request — should  be  used.  In  addition,  this  study  represents  less  than 
80  man-hours  of  effort— primarily  by  analysts  not  in  the  fixed-wing  aircraft  survivability 
community.  It  is  indisputable  that  a  dedicated  effort  by  appropriately  placed  analysts  would  derive 
data  that  differ  somewhat  from  those  presented  here;  such  data  may  lead  to  conclusions  that  are 
quantitatively  different  from  those  presented  in  this  study. 

It  must  also  be  noted,  however,  that  the  exposure  of  all  data  and  assumptions,  as  presented  here  and 
in  the  classified  supplement,  is  an  integral  part  of  the  knowledge-based  benefit/cost  methodology.  Thus, 
any  differences  in  conclusions  will  be  directly  relatable  to  differences  in  specific  assumptions  and  data. 
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In  general,  such  differences  are  testable.  Thus,  in  principle,  a  determined  application  of  the 
knowledge-based  benefit/cost  methodology  must  converge  to  a  single  set  of  conclusions. 


Background 

In  1981,  a  manual  analysis  of  the  vulnerability  of  the  conceptual  F-16  was  accomplished  by  analysts 
at  the  Aeronautical  Systems  Division,  Wright-Patterson  Air  Force  Base  (WPAFB)  (Lentz,  Bennett,  and 
Cramer  1981).  The  availability  of  this  evaluation  and  its  similarity  to  those  that  could  have  been  done 
in  the  MS  n  time  frame  of  the  F-16  made  it  an  ideal  basis  for  applying  knowledge-based  benefit/cost 
methodology.  The  evaluation  considered  the  six  principle  (orthogonal)  aspects  of  an  aircraft,  which 
encompassed  the  characteristics  of  the  F-16  as  understood  at  the  time  of  analysis.  For  each  aspect,  the 
silhouette  of  the  aircraft  was  drawn.  Superimposed  upon  the  silhouette  were  the  regions  which  contained 
critical  components  of  the  aircraft.  These  regions  were  identified  by  the  salient  critical  component 
contained  (see  Tables  13—17).  However,  each  region  may  have  included  additional  critical  components. 
An  example  is  the  cockpit  region,  which,  in  addition  to  the  pilot,  contains  other  flight-critical  components 
not  specifically  listed  in  the  table. 

The  presented  area  of  each  region  was  determined  by  reference  to  drawings,  similar  components  in 
other  aircraft,  and  analysts  expertise.  In  addition,  the  presented  area  ascribed  to  each  region  was 
expanded  to  account  for  the  fragment  spray  and  blast  region  associated  with  a  23-mm  HEI  projectile. 

In  1984,  subsequent  to  the  previous  analysis,  the  JLF  test  series  began.  In  response,  a  plan,  for  the 
testing  of  aircraft  was  developed  by  the  Joint  Technical  Coordinating  Group  on  Aircraft  Survivability 
(JTCG/AS).  In  a  1984  report  (JLF  1984),  an  extensive  series  of  component  and  subsystem  level  tests  are 
proposed,  including  a  cost  estimate  for  testing  categorized  into  the  following  subsystems: 

•  Fuel  (including  fuel  ingestion  damage). 

•  Propulsion  (including  fuel  ingestion  tolerance). 

•  Flight  Controls. 
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Table  13.  Critical  Regions,  Top 


Region  Number 

Major  Vulnerable  Component 

Associated  Subsystem 

T1 

Pilot 

Crew  Station 

T2 

Ammunition  Drum 

Ammunition  Drum 

T3 

night  Control  Computer 

Misc.  Flight  Instruments 

T4 

Hydraulic  Reservoirs 

Hyd.  Res.,  Acc.,  and  Lines 

T4A 

Hydraulic  Lines 

Hyd.  Res.,  Acc.,  and  Lines 

T5 

Leading  Edge  Actuator 

Actuators  and  Lines 

T6 

Flaperon  Actuators 

Actuators  and  Lines 

T7 

Horizontal  Tail  Actuators 

Actuators  and  Lines 

T8 

F-1  Fuel  Tank 

FI  Fuel  Tank 

T9 

F-2  Fuel  Tank 

F2  Fuel  Tank 

TIO 

Reservoir  Tanks 

Reservoir  Fuel  Tank 

Til 

A-1  Fuel  Tank 

A1  Fuel  Tank  and  Lines 

Til  A 

Fuel  Feed  Line 

A1  Fuel  Tank  and  Lines 

T12 

Engine 

Engine  and  Accessories 

Table  14.  Critical  Regions,  Bottom 


Region  Number 

Major  Vulnerable  Component 

Associated  Subsystem 

B1 

PUot 

Crew  Station 

B2 

Flight  Control  Computer 

Misc.  Flight  Instruments 

B3 

Pitch  Rate  Gyro 

Misc.  Flight  Instruments 

B4 

Normal  Accelerometer 

Misc.  Flight  Instruments 

B5 

Leading  Edge  Actuator 

Actuators  and  Lines 

B6 

Hydraulic  Reservoirs 

Hyd.  Res.,  Acc.,  and  Lines 

B7 

Flaperon  Actuators 

Actuators  and  Lines 

B8 

Horizontal  Tail  Actuators 

Actuators  and  Lines 

B8A 

Hydraulic  Lines 

Actuators  and  Lines 

B8B 

Bleed  Air  Lines 

Actuators  and  Lines 

B9 

F-1  Fuel  Tank 

FI  Fuel  Tank 

BIO 

F-2  Fuel  Tank 

F2  Fuel  Tank 

Bll 

Reservoir  Tanks 

Reservoir  Fuel  Tank 

B12 

A-1  Fuel  Tank 

A1  Fuel  Tank  and  Lines 

B12A 

Fuel  Feed  Line 

A1  Fuel  Tank  and  Lines 

B13 

Engine  Accessories 

Engine  and  Accessories 
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Table  15.  Critical  Regions,  Left  Side 


Region  Number 

Major  Vulnerable  Component 

Associated  Subsystem 

LSI 

Pilot 

Crew  Station 

LS2 

Flight  Control  Computer 

Misc.  Flight  Instruments 

LS3 

Normal  Accelerometer 

Misc.  Flight  Instruments 

LS4 

Leading  Edge  Actuator 

Actuators  and  Lines 

LS5 

Flaperon  Actuator 

Actuators  and  Lines 

LS6 

Rudder  Actuators 

Actuators  and  Lines 

LS7 

Hydraulic  Accumulators 

Hyd.  Res.,  Acc.,  and  Lines 

LS7A 

Hydraulic  Lines 

Hyd.  Res.,  Acc.,  and  Lines 

LS7B 

Bleed  Air  Lines 

Hyd.  Res.,  Acc.,  and  Lines 

LS8 

Ammunition  Drum 

Ammunition  Drum 

LS9 

F-1  Fuel  Tank 

FI  Fuel  Tank 

LSIO 

F-2  Fuel  Tank 

F2  Fuel  Tank 

LSll 

Reservoir  Tanks 

Reservoir  Fuel  Tank 

LS12 

A-1  Fuel  Tank 

A1  Fuel  Tank  and  Lines 

LS13 

Engine 

Engine  and  Accessories 

LS14 

Engine  Accessories 

Engine  and  Accessories 

Table  16.  Critical  Regions,  Right  Side 


Region  Number 

Major  Vulnerable  Component 

Associated  Subsystem 

RSI 

Pilot 

Crew  Station 

RS2 

Flight  Control  Computer 

Misc.  Flight  Instruments 

RS3 

Normal  Accelerometer 

Misc.  Flight  Instruments 

RS4 

Pitch  Rate  Gyro 

Misc.  Flight  Instruments 

RS5 

Leading  Edge  Actuator 

Actuators  and  Lines 

RS6 

Flaperon  Actuator 

Actuators  and  Lines 

RS7 

Rudder  Actuators 

Actxiators  and  Lines 

RS8 

Hyd.  Acc.  -  Vertical  Tail 

Hyd.  Res.,  Acc.,  and  Lines 

RS8A 

Hydraulic  Lines 

Hyd.  Res.,  Acc.,  and  Lines 

RS9 

Ammunition  Drum 

Ammunition  Drum 

RSIO 

F-1  Fuel  Tank 

FI  Fuel  Tank 

RSll 

F-2  Fuel  Tank 

F2  Fuel  Tank 

RS12 

Reservoir  Tanks 

Reservoir  Fuel  Tank 

RS13 

A-1  Fuel  Tank 

A1  Fuel  Tank  and  Lines 

RS13A 

Fuel  Feed  Line 

A1  Fuel  Tank  and  Lines 

RS14 

Engine 

Engine  and  Accessories 

RS15 

&igine  Accessories 

Engine  and  Accessories 
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Table  17.  Critical  Regions,  Front  and  Rear 


Region  Number 

Major  Vulnerable  Component 

Associated  Subsystem 

FI 

Pilot 

Crew  Station 

R2 

Rudder  Actuators 

Actuators  and  Lines 

R3 

Engine 

Engine  and  Accessories 

•  Hydraulics. 

•  Structure. 

•  Armament. 

•  Crew  Station. 

•  MiscellaneousAJnique. 

•  Rotor/Drive  Train. 

The  previously  referenced  reports  (Lentz  1981;  JTCG/AS  1984)  formed  the  basis  for  the  data  used 
in  this  example  application.  In  addition,  significant  help  was  provided  by  Mr.  Ralph  Lauzze,  WPAFB; 
Ms.  Lisa  Woods,  Booz-Allen  Hamilton,  Inc.;  and  Mr.  Robert  Walther,  Dr.  James  Walbert,  and 
Mr.  William  Keithley,  ARUSLAD. 

5.3.2  Application  of  Mefliodology. 

Quality  of  Vulnerability  Information 

Tables  13-17  list  the  66  regions,  over  six  aspects,  that  were  deemed  critical  (Lentz  1981).  In  the 
original  (as  reproduced  in  the  classified  supplement  to  this  report),  each  region  has  an  associated  "Adjusted 
Presented  Area,"  "Average  Region  (Pk/h),"  and  a  "Vulnerable  Area,"  which  was  the  product  of  the  first 
two  values.  These  values  were  removed  here  to  allow  an  unclassified  presentation  of  this  work.  However, 
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it  is  important  to  recognize  their  existence,  since  it  was  the  "Average  Region  (Pk/h)"  that  formed  the  basis 
of  the  benefit  portion  of  this  example. 

Tables  13-17  also  contain  a  column  entitled  "Associated  Subsystem."  This  association,  made  as  part 
of  this  example,  wiU  be  explained  next. 

Before  proceeding  with  this  example,  it  is  important  to  understand  the  "manual"  analysis  described 
in  the  JLF  document  (JTCG/AS  1984).  For  each  of  the  regions  in  Tables  13-17,  the  vulnerability  of  all 
components  in  the  region  were  considered.  The  analysts  used  their  expertise  to  gauge  the  probability  of 
component  damage  given  a  hit  in  the  region,  from  the  Usted  aspect,  by  a  23-mm  HEI  round.  The  analysis 
included  assessment  of  the  standoff  (the  distance  between  the  probable  detonation  point  and  each 
component)  and  the  probable  shielding,  as  well  as  the  susceptibility  of  the  components  themselves.  Also 
included,  in  a  rather  subjective  way,  was  an  allowance  for  die  chance  that  an  unforeseen  event  would  take 
place;  this  allowance  was  based  upon  the  complexity  of  the  region  as  well  as  upon  the  expert  judgment 
of  the  analysts.  This  information  was  all  rolled  up  into  a  single  number  for  each  region,  the  "Average 
Region  (Pk/h).  To  determine  the  Pk/h,  the  probability  of  target  kill  given  a  hit  on  the  region  attributable 
to  each  major  component  in  the  region  was  estimated.  If  one  component  dominated  the  region,  its  kill 
probability  was  used  as  the  region  Pk/h;  or  else,  an  average  of  the  kUl  probabilities  of  the  major 
components  was  used.  The  final  value  was  then  adjusted  (upward)  to  accoimt  for  synergistic  effects  (i.e., 
those  target  kiUs  that  cannot  be  attributed  to  a  single  component). 

The  present  knowledge-based  benefit/cost  analysis  focused  on  the  uncertainties  associated  with  the 
"Average  Region  (Pk/h)."  To  quantify  the  benefit  of  various  test  plans,  an  adaptation  of  the  "Fuzzy  Math" 
technique  was  used.  Using  an  expert  from  the  Air  Systems  Branch  (ASB)  of  the  ARL,  the  Pk/h  assigned 
to  each  entry  in  Tables  13-17  was  "Fuzzified"  (i.e.,  each  single  value  was  replaced  by  two  ranges  of 
values:  one  range  in  which  the  real  Pk/h  probably  lies— as  judged  by  the  ASB  expert,  and  a  larger, 
encompassing  range  in  which  the  value  most  certainly  lies).  These  ranges  correspond  to  the  50% 
confidence  interval  and  the  99%  confidence  interval  described  in  the  preceding  sections  of  this  report. 
These  ranges  were  then  propagated  through  the  manual  methodology,  resulting  in  a  "Fuzzy"  total 
vulnerable  area  (i.e.,  a  vulnerable  area  expressed  as  a  50%  confidence  interval  and  a  99%  confidence 
interval).  These  were  combined  into  a  single  80%  confidence  interval  as  described  in  the  preceding 
sections. 
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In  addition  to  placing  a  range  of  uncertainty  around  each  Pk/h,  the  ASB  expert  was  also  asked  to  thuik 
about  the  elements  in  each  region  that  contributed  to  the  uncertainty.  With  that  in  mind,  the  analyst 
defined  two  sets  of  tests  to  address  that  uncertainty,  one  involving  only  component  and  subsystem  tests, 
and  one  involving  full-up  LFT.  Finally,  the  analyst  estimated  the  reduction  in  uncertainty  (expressed  as 
reduced  sizes  for  the  confidence  intervals)  given  that:  (a)  the  subsystem  tests  alone  were  completed,  and 
(b)  that  the  full-up  LFT  was  done  in  addition  to  the  subsystem  tests. 

To  simplify  the  previous  procedure,  it  was  noted  that,  although  Tables  13-17  contain  66  entries,  the 
regions  and  associated  components  could  be  placed  into  10  categories.  These  are  listed  in  Table  18. 
Table  18  also  lists  the  number  of  shots  included  in  each  test  category  in  order  to  achieve  the  reduction 
in  uncertainty  assigned  by  the  ASB  expert. 

At  this  point,  it  is  important  to  note  the  coarseness  of  the  test  solutions  presented  in  this  example. 
Clearly,  there  are  an  infinite  number  of  possible  shots;  hence  the  reduction  to  only  three  possibilities  (a 
fixed  subsystem  series,  a  fixed  LFT  series  plus  the  subsystem  series,  or  no  tests  at  all)  is  obviously  a 
simplification.  However,  recall  that  the  purpose  of  this  example  is  to  demonstrate  the  feasibility  of  the 
method.  Just  as  the  use  of  aeronautical  engineers  from  the  manufacturer  of  the  F-16  was  unwarranted  for 
this  example,  so  is  the  consideration  of  a  fine  tuning  of  the  test  series.  Moreover,  as  shown  next,  the 
59,049  combinations  of  test  possibilities  evaluated  in  this  study  were  significantly  more  than  would 
normally  be  considered. 

With  the  above  data,  the  80%  confidence  interval  associated  with  the  total  vulnerable  area  was  found 
to  be: 


•  0.359  (35.9%)  -  Baseline  (no  testing) 

•  0.199  (19.9%)  -  All  subsystem  test  series  done;  no  LFT 

•  0.120  (12.0%)  -  All  test  series  (LFT  and  subsystem)  done. 
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Table  18.  Test  Categories 


Test  Category 

Shots 

Subsystem  Series 

Shots 

Live-Fire  Series 

Crew  Station 

8 

4 

Ammunition  Drum 

5 

2 

Misc.  night  Instruments 

8 

4 

Hyd.  Res.,  Acc.,  and  Lines 

6 

3 

Actuators  and  Lines 

8 

4 

F-1  Fuel  Tank 

10 

5 

F-2  Fuel  Tank 

10 

5 

Reservoir  Fuel  Tank 

10 

5 

A1  Fuel  Tank  and  Lines 

10 

5 

Engine  and  Accessories 

6 

3 

At  this  point,  the  major  set  of  calculations  were  performed.  All  combinations  (59,049)  of  the  three 
possibilities  (a  fixed  subsystem  series,  a  fixed  LPT  series  plus  the  subsystem  series,  or  no  tests  at  all)  for 
each  of  the  10  test  categories  were  considered,  and  the  80%  confidence  interval  associated  with  each 
combination  was  computed.  These  were  combined  with  the  associated  costs  as  described  next. 

Costs 

Equally  specialized  is  the  area  of  assigning  costs  to  the  various  options.  The  following  section 
describes  the  method  used  for  this  example.  However,  it  must  be  noted  that,  were  this  an  actual  live-fire 
waiver  analysis,  many  resources  would  be  used  which  were  not  invoked  for  this  example.  Again,  the 
purpose  of  this  example  is  to  demonstrate  the  feasibility  of  the  method. 

It  must  also  be  noted  that,  in  making  estimates,  one  generally  envisions  a  range  of  values  rather  than 
a  crisp  number.  (That  is,  in  fact,  the  basis  of  the  "Fuzzy  Analysis"  described  in  the  preceding  sections.) 
The  estimation  of  costs,  likewise,  leads  to  a  range  of  values  and  estimation  techniques.  In  such  a  case, 
it  is  common  to  consciously  skew  all  estimates  to  favor  one  course  of  action  so  that — ^if  that  course  of 
action  should  still  prove  cost-ineffective — ^the  coiuse  may  confidently  be  excluded.  In  this  example,  we 
have  skewed  the  estimates  to  favor  the  LFT  series.  Thus,  in  those  cases  in  which  the  LFT  series  prove 
cost-ineffective,  the  result  will  not  depend  upon  a  subjective  assignment  that  was  unfavorable  to  LFT. 
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For  the  estimation  of  subsystem  test  series  costs,  adjustments  to  an  existing  set  of  estimates  were 
made.  As  listed  previously,  JTCG/AS  (1984)  contains  the  estimates  made  in  1984  for  subsystem  level 
test  series  for  the  F-16.  These  costs  include: 

•  Program  planning. 

•  Design  of  replica  targets. 

•  Procurement  or  manufacture  of  replica  targets. 

•  Pretest  preparation  of  test  articles. 

•  Conduct  of  the  tests. 

•  Munitions/weapons. 

•  Data  acquisition  and  reduction. 

•  Documentation. 

•  Unique  administrative  costs. 

The  subsystem  level  tests  from  JTCG/AS  (1984)  were  easily  mapped  onto  the  test  categories  listed 
in  Table  18.  Thus,  a  baseline  set  of  costs  was  available.  These  costs,  for  each  test  category,  were  then 
increased  in  two  ways.  First,  the  cost  of  the  subsystem  test  series  in  each  category  was  multiplied  by  a 
factor  (1.34)  to  convert  1984  costs  to  a  common  (1993)  base  (Norton  1993).  In  addition,  although 
"conduct  of  tests"  and  "munitions/weapons"  were  included  in  die  JTCG/AS  (1984)  costs,  the  number  of 
shots  in  each  series  was  not  given.  Hence,  in  order  to  account  for  the  particular  number  of  shots 
prescribed  by  the  ASB  expert  to  achieve  the  associated  uncertainty  reduction,  an  additional  $25k  per  shot 
was  added  to  the  already  specified  costs.  Although  this  procedure  may  be  "double-counting"  the  expense 
associated  with  each  shot,  it  is  m  keeping  with  the  LFT-favoring  skew  described  previously. 

The  values  used  in  this  example  are  listed  in  Table  19. 
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Table  19.  Subsystem  Costs 


Test  Category 

Subsystem  Series  Cost 
($) 

Crew  Station 

570k 

Ammunition  Drum 

295k 

Miscellaneous  Flight  Instruments 

1.2k 

Hyd.  Res.,  Acc.,  and  Lines 

685k 

Actuators  and  Lines 

434k 

FI  Fuel  Tank 

2.85k 

F2  Fuel  Tank 

2.85k 

Reservoir  Fuel  Tank 

2.85k 

A1  Fuel  Tank  and  Lines 

2.85k 

Engine  and  Accessories 

850k 

In  evaluating  the  LFT  costs,  it  is  essential  to  recall  that  the  knowledge-based  benefit/cost  methodology 
being  applied  here  specifies  the  use  of  expected  costs,  as  described  in  the  preceding  sections.  Calculation 
of  expected  costs  requires  estimates  of  probable  outcomes  of  firing,  as  well  as  the  costs  associated  with 
each  outcome.  In  this  case,  two  possibilities  were  considered  for  each  test  category.  First,  the  probability 
of  total  loss  of  the  target  was  assessed.  For  this  assessment,  the  probability  of  total  loss  of  the  target, 
given  that  the  region  was  killed,  was  assessed  to  be  either  high  (0.8),  medium  (0.4),  or  low  (0.02).  Then 
the  probability  of  total  loss  from  any  shot  was  taken  to  be  the  Pk/h  for  the  region  times  the  high,  medium, 
or  low  resultant  total  loss  probability. 

If  the  target  was  a  total  loss,  the  cost  of  the  test  was  assessed  to  be  the  cost  of  the  target  plus  a  $50k 
assessment  for  the  cost  of  an  LFT  shot.  In  keeping  with  the  LFT  skew,  the  cost  of  the  target  was  taken 
as  the  production  cost  as  presented  by  Lauzze  and  Griffis  (1993).  * 

If  the  target  is  not  lost,  the  cost  of  an  LFT  was  assessed  to  be  the  cost  of  the  shot  ($50k)  plus  the  cost 
to  return  the  target  to  its  preshot  condition.  Based  upon  extensive  LFT  done  by  the  ARL,  multiplied  by 
the  relative  cost  of  an  F-16  to  the  costs  of  the  ARL  targets,  a  figure  of  $400k  was  assessed  for  the  post 
shot  repair  to  be  applied  to  aU  nontotal-loss  shots. 

The  total  loss  probabilities  and  resulting  expected  costs  used  in  this  example  are  listed  in  Table  20. 
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Table  20.  Expected  Cost  Data 


Test  Category 

Total  Loss  Probability 

Live-Fire  Series  Cost 
($) 

Crew  Station 

0.02 

■■iH 

Ammunition  Drum 

0.64 

Miscellaneous  Flight  Instruments 

0.02 

3.6M 

Hydraulic  Reservoirs,  Accumulators,  and  Lines 

0.36 

26.2M 

Actuators  and  Lines 

0.02 

3.6M 

FI  Fuel  Tank 

0.56 

65.6M 

F2  Fuel  Tank 

0.70 

81.2M 

Reservoir  Fuel  Tank 

0.66 

76.8M 

A1  Fuel  Tank  and  Lines 

0.66 

76.8M 

Engine  and  Accessories 

0.32 

24.0M 

Using  this  data,  the  expected  costs  for  all  59,049  combinations  of  the  three  possibilities  (a  fixed 
subsystem  series,  a  fixed  LET  series  plus  the  subsystem  series,  or  no  tests  at  all)  for  each  of  the  10  test 
categories  were  calculated. 

5.3.3  Results.  For  completeness,  it  is  necessary  that  this  example  be  carried  through  to  the  analysis 
of  results.  However,  as  stressed  previously,  the  values  used  in  this  example  were  obtained  without 
recourse  to  many  of  the  resources  that  would  be  available  to  a  program  manager  who  would  be  evaluating 
the  advisability  of  an  LFT  waiver.  In  addition,  this  example  only  discusses  one  of  many  possible  threat 
munitions.  Hence,  the  application  of  these  results  to  the  actual  test  plans  for  the  F- 16  is  certainly  not 
warranted. 

Clearly,  the  presentation  of  59,049  sets  of  benefit/cost/test  plan  triads  is  problematic.  To  cull  out  the 
best  options,  a  cost-effectiveness  figure,  defined  as  confidence-interval  reduction  per  dollar,  was  calculated 
for  each  of  the  59,049  test  plans.  The  resulting  sets  of  cost-effectiveness/benefit/cost/test  plan  were  then 
sorted  in  order  of  most  to  least  cost-effective.  From  this  list,  it  is  an  easy  matter  to  extract  the  most 
cost-effective  means  of  achieving  each  level  of  certainty  in  the  vulnerability  data.  These  most  effective 
test  plans  are  presented  in  Table  21  results  and  are  plotted  vs.  cost  in  Figure  18.  In  Table  21,  the  test  plan 
is  presented  as  a  sequence  of  10  digits,  indicating  the  test  status  of  each  of  the  ten  test  categories  as  listed 
in  Tables  18,  19,  and  20.  For  example,  test  plan  2222212223  consists  of  subsystem  level  tests  ("2")  for 
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Figure  18.  Cost  vs.  80%  confidence  interval. 
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Table  21.  Most  Effective  Test  Plans 


Width  of  80%  Conf. 
Int. 

Cost-Effectiveness 

Cost  ($M) 

Test  Plan 

.359 

0.00 

1111111111 

.30 

0.0211 

3.42 

2111111121 

.29 

0.0211 

3.42 

2111111121 

.28 

0.0173 

4.56 

2211111122 

.27 

0.0172 

6.27 

2111111221 

.26 

0.0172 

6.27 

2111111221 

.25 

0.0166 

6.56 

2211111221 

.24 

0.0142 

9.12 

2111112221 

.23 

0.0142 

9.12 

2111112221 

.22 

0.0130 

10.70 

2211212222 

.21 

0.0124 

11.97 

2111122221 

.20 

0.0112 

14.23 

2212222222 

.19 

0.0064 

26.23 

3232322222 

.18 

0.0023 

76.43 

3233322223 

.17 

0.0019 

99.43 

3122322232 

.16 

0.0016 

127.03 

3232322233 

.15 

0.0011 

195.43 

3212222333 

.14 

0.0009 

261.03 

3212232333 

.13 

0.0007 

307.63 

3223323333 

.12 

0.0007 

361.70 

3233333333 

all  test  categories  except  category  6  (F-1  fuel  tank),  which  is  not  tested  ("1"),  and  category  10  (engines 
and  accessories),  for  which  both  subsystem  and  LFT  are  done  ("3"). 

For  the  remainder  of  this  discussion,  we  win  use  the  80%  confidence  interval,  listed  in  Table  21  and 
plotted  in  Figure  18,  as  a  measure  of  the  uncertainty  in  the  vulnerable  area,  the  information  being  used 
in  decisions  (productions,  modifications,  tactics,  etc.)  regarding  the  subject  aircraft.  The  choice  and 
definition  of  the  80%  confidence  interval  as  the  measure  of  uncertainty  was  an  analyst’s  decision.  Clearly, 
the  "Fuzzy"  outputs  from  the  analysis  are  a  direct,  detailed  measure  of  the  imcertainty;  the  choice  of  one 
particular  confidence  interval,  based  upon  those  Fuzzy  outputs,  is  largely  a  matter  of  taste. 

Note,  also,  that  the  chosen  confidence  interval  encompasses  the  whole  range.  That  is,  an  uncertainty 
of  20%  corresponds  to  ±10%. 
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Were  this  an  actual  benefit/cost  analysis,  the  analyst  would  now  be  in  a  position  to  rate  the  various 
test  plans  and  to  draw  some  conclusions  regarding  which  tests  should  be  done.  For  example.  Figure  18 
shows  that  reducing  the  uncertainty  in  the  vulnerable  area  down  to  19-20%  is  quite  cost-effective. 
Referring  to  Table  21,  the  reduction  in  the  imcertainty  per  dollar  is  in  the  order  of  1.0%  per  million 
dollars.  (The  reduction  from  0.359,  the  starting  uncertainty,  to  0.2  could  be  accomplished  with  a  test  plan 
costing  $14.2M.) 

However,  reductions  below  19%  become  very  expensive.  For  example,  the  cost  of  reducing 
uncertainty  to  15%  is  $195M;  the  payoff  (reduction  in  uncertainty)  is  only  0.1%  per  million  dollars.  A 
more  critical  view  is  to  look  at  the  differential  cost  of  the  reduction  below  20%:  To  gain  the  extra  5% 
reduction  in  uncertainty  costs  $181.2M  ($195.4-14.2M)  or  a  payoff  of  only  0.03%  per  mUlion  dollars. 
The  reason  for  this  behavior  is  clear.  In  this  example,  uncertainty  reductions  below  19%  require 
increasing  amounts  of  LFT,  which  cany  a  relatively  high  expected  cost. 

The  area  around  19%  uncertainty  is,  in  this  example,  a  grey,  transitional  area.  It  is  not  possible  to 
approach  19%  reduction  without  any  LFT:  the  Test  Plan  2222222222,  comprising  all  subsystem  tests, 
only  provides  a  reduction  to  19.95%.  In  addition.  Test  Plan  2222222222  requires  the  detailed  subsystem 
testing  of  components  such  as  the  miscellaneous  flight  instruments,  which  have  a  low  presented  area  and 
high  cost.  As  a  result,  the  differential  cost  of  improvement  from  20%  to  19.9%  is  $1.2M  or  $0.08  per 
million  dollars.  Thus,  should  small  improvement  below  20%  be  required,  it  is  equally  cost-effective  to 
conduct  limited  fuU-up  LFT  on  high  profile  areas  that  have  a  relatively  low  expected  cost.  For  example, 
shots  into  the  crew  station  have  a  relatively  low  probability  of  causing  complete  loss  of  the  target.  Such 
shots  may  address  many  of  the  "unknown-unknowns"  and  were  thus  assessed  by  the  ASB  expert  as 
providing  moderate  reduction  in  the  uncertainty  of  the  crew  station  Pk/h  beyond  those  provided  by 
subsystem  tests.  Since  the  crew  station  also  represents  a  significant  firaction  of  the  critical  presented  area 
from  five  of  the  six  aspects,  it  is  reasonable  to  expect  LFT  of  this  region  to  be  among  the  first  to  enter 
into  a  cost-effective  test  plan. 

On  the  other  hand,  note  that  LFT  of  the  ammunition  drum,  in  this  example,  never  appears  in  Table  21. 
For  this  region,  the  low  presented  area  and  high  probability  of  total  target  loss  outweigh  the  gain  in 
uncertainty:  such  tests  are  not  cost-effective. 
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The  previous  observations  provide  insight  to  the  reasons  for  the  results  that  evolved  in  this  example. 
The  dominant  factor  in  the  cost-effectiveness  of  a  test  plan  is  its  expected  cost.  This,  in  turn,  is  driven 
by  the  probability  of  total  system  loss  and  the  cost  of  a  system.  Thus,  a  similar  analysis  on  a  less  costly 
system  should  show  LFT  to  be  more  cost-effective  than  was  shown  here.  Similarly,  shots  on  a  system 
that  are  unlikely  to  cause  total  loss  tend  to  have  a  markedly  lower  expected  cost  and  thus  higher 
cost-effectiveness.  The  complexity  of  the  region  under  attack  also  plays  an  influential  role.  A  complex 
region  is  generally  felt  to  contain  more  opportunities  for  "unknown-unknowns"  and,  hence,  realize  more 
uncertainty  reductions  from  LFT.  And,  finally,  the  size  of  a  region  plays  the  role  of  an  overall  weighting 
factor,  since  it  appears  as  a  multiplier  in  all  outputs  related  to  vulnerable  area.  Thus,  a  relatively 
inexpensive  target,  whose  complex  regions  are  large  and  unlikely  to  be  catastrophically  killed,  may  be  a 
prime  candidate  for  LFT  tests. 

5.3.4  Conclusions.  IT  MUST  AGAIN  BE  NOTED  THAT  THIS  EXAMPLE  DOES  NOT 
CONSTITUTE  AN  ACTUAL  LFT  WAIVER  ANALYSIS. 

The  ptirpose  of  this  exercise  was  to  scope  the  level  of  difficulty  to  be  encountered  in  the  application 
of  the  knowledge-based  benefit/cost  methodology  to  an  actual  system.  As  stated  in  the  introduction  to 
this  section,  this  example  represents  less  than  80  man-hours  of  effort — ^primarily  by  analysts  not  in  the 
fixed-wing  aircraft  survivability  community.  In  an  actual  LFT  waiver  analysis,  such  resources  as  are 
available  to  the  system  project  manager  would  be  brought  to  bear.  While  the  level  of  expertise  and 
availability  of  knowledge  would  therefore  be  higher,  it  is  also  true  that  the  level  of  detail — especially  in 
the  analysis  of  vulnerability  data  and  the  diversity  of  potential  test  plans — ^would  also  be  higher.  Thus, 
while  we  conclude  that  application  of  the  knowledge-based  benefit/cost  methodology  to  an  actual  system 
is  certainly  tractable,  we  anticipate  that  the  expenditure  of  only  80  man-hoiurs  is  probably  a  lower  bound. 

This  example,  in  conjunction  with  the  Ballistic  Bicycle  example  presented  in  preceding  sections, 
demonstrates  the  kinds  of  conclusions  that  can  be  supported  through  a  knowledge-based  benefit/cost 
analysis. 

The  inputs  to  the  analysis  were  technical  judgments,  based  upon  previous  test  experience  with  other 
targets,  details  of  the  subject  target  which  would  be  emerging  at  MS  n,  and  tests  on  the  subject  target 
conducted  prior  to  the  analysis.  These  inputs  included: 
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•  Available  MS  II  vulnerability  analyses. 

•  Estimates  of  uncertainties  in  the  inputs  to  such  analyses. 

•  Conceptual  test  plans,  geared  to  address  those  uncertainties. 

•  Estimated  costs  of  those  test  plans,  including  probable  outcomes  of  each  plan. 

The  outputs  in  this  example  included  total  and  differential  gains  accruing  from  many  possible  test 
plans. 

In  the  original  statement  of  the  knowledge-based  benefit/cost  methodology,  the  requirement  for  a  given 
level  of  confidence  in  the  vulnerability  information  was  to  be  met  by  an  assessment  of  the  decision 
processes  which  use  the  resultant  vulnerability  information.  That  is,  a  goal  in  uncertainty  reduction  was 
to  be  established  a  priori  and  test  plans  formulated  to  meet  that  goal.  This  example  illustrates  another 
approach  to  the  justification  of  test  plan  selection.  Figiure  18  provides  a  graphical  realization  of  the  cost 
of  certainty  in  vulnerability  information.  Having  conducted  such  an  analysis,  one  is  in  a  solid  position 
to  demonstrate  the  marginal  gain  vs.  cost,  thus  imposing  reasonable  limitations  to  the  expectations  of  the 
decision  makers. 

5.4  Aircraft  Attrition-Based  Methodology.  Section  5.4  is  provided  by  Mr.  Ralph  W.  Lauzze  n  of  the 
Vehicle  Subsystems  Division  (WL/FIV),  USAF,  WPAFB,  OH. 

Reducing  combat  system  losses  (attrition)  is  a  primary  objective  of  any  aggressive  vulnerability 
reduction  program.  If  the  results  of  LFT  can  be  shown  to  directly  result  in  reduced  attrition,  then  that 
savings  can  be  a  useful  measure  of  "benefit"  in  a  cost-benefit  assessment  for  LFT.  A  relatively  simple 
attrition-based  methodology,  comparing  estimated  savings  due  to  reduced  losses  with  the  estimated  cost 
of  executing  an  LFT  program,  has  been  developed.  The  methodology  has  been  applied  to  sample  cases 
for  aircraft  systems  and  may  have  utility  for  evaluating  other  vehicle  classes.  The  methodology  assumes 
a  previously  undetected  vulnerability  "flaw"  exists  in  die  system.  This  causes  the  system’s  true 
vulnerability  to  be  higher  than  originally  assessed.  It  is  further  assumed  that  the  flaw  is  discovered  during 
LFT  and  subsequently  fixed  through  redesign  (and  retrofit,  if  necessary).  By  comparing  the  "before"  and 
"after"  vulnerabilities,  an  estimated  savings  in  attrition  can  be  calculated.  This  savings  is  then  compared 
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with  the  total  cost  of  performing  the  LFT  program,  including  hardware,  test,  redesign,  and  retrofit  costs. 
The  methodology  is  flexible,  allowing  comparisons  to  be  made  for  the  estimated  payoff  resulting  from  the 
many  variations  and  combinations  of  options  to  vulnerability  assessment  and  reduction,  ranging  from 
analysis-only  to  full-up  system-level  LFT.  The  sensitivity  of  the  payoff  to  the  various  input  estimates  can 
easily  be  made,  allowing  the  analyst  to  examine  a  whole  range  of  inputs.  This  can  be  particularly  useful 
where  confidence  in  input  estimates  is  lacking.  This  overall  approach  to  cost-benefit  assessment  can  be 
beneficial  to  the  plarmer  and  analyst  in  examining  where  resources  could  best  be  applied  to  assure  the 
most  payoff  for  the  LFT  investment.  It  may  also  be  useful  to  examine  the  efficacy  of  pursuing  a  waiver 
for  system-level  LFT. 

Several  parameters  are  not  currently  covered  in  the  methodology,  but  are  being  addressed.  An 
important  payoff  from  LFT  is  improved  confidence  levels  in  the  subject  system’s  vulnerability  or  lethality. 
While  not  currently  included  in  the  methodology,  a  set  of  subjective  confidence  factors  could  be 
developed,  based  on  the  specific  test  options  of  interest,  and  assigned  to  the  respective  cost-benefit 
calculation  in  order  to  "weight"  each  approach.  The  cost  of  crew  casualties  is  not  currently  included  in 
the  methodology;  however,  estimates  could  also  be  developed  for  incorporation.  Also,  the  impact  of 
Aircraft  Battle  Damage  and  Repair  (ABDR)  is  not  currently  included  in  the  methodology.  The  costs 
associated  with  repairing  damaged  aircraft  and  the  impact  of  aircraft  that  are  repaired  and  returned  to 
combat  could  also  be  included.  Additionally,  the  impact  of  improved  operational  effectiveness  due  to 
increased  system  robustness  (i.e.,  more  tanks  killed  for  fewer  system  losses,  etc.)  is  not  currently,  but 
could  be,  included. 

The  methodology  consists  of  the  following  steps: 

5.4.1  Select  a  Representative  Sortie/Threat  Scenario.  Develop  an  estimate  of  the  total  number  of 
combat  sorties,  number  of  aircraft  involved,  and  an  "allowable"  attrition  rate  (Probability  of  Kill  given  a 
Sortie  -  Pk/s).  This  data  is  generally  available  from  the  plaimed  use  of  the  aircraft  and  can  be  compared 
with  similar  systems  in  prior  conflicts  to  gain  a  historical  perspective. 

5.4.2  Determine  Production/Cost  Data  for  New  Aircraft.  From  manufacturing/production  plans, 
determine  the  production  rate  and  production  cost  of  the  system  to  be  tested.  Data  on  the  system’s 
production  rate  is  needed  for  cases  where  retrofit  is  a  potential  result  firom  the  LFT  program.  Data  on  the 
production  cost  of  the  system  is  needed  to  evaluate  replacement  costs  for  systems  lost  due  to  combat 
attrition. 
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5.4.3  Establish/Detennine  Vulnerability  Baseline  Data.  From  the  system’s  specifications,  determine 
Pk/h  information  for  the  "allowable"  attrition  for  the  threat(s)  of  interest.  Determine,  for  kill  levels  of 
interest  -  generally  "K"  (loss  of  control  within  30  s),  "A"  (loss  of  control  within  5  min),  "B"  (loss  of 
control  within  30  min),  and  "M"  (prevention  of  completing  the  mission,  but  not  lost  from  the  inventory), 

5.4.4  Calculate  Estimated  Hit  Rate  (Ph/s).  An  estimate  of  the  Ph/s  can  be  developed  from  the 
allowable  attrition  rate  and  the  spec  values  for  Pk/h.  (Pk/s  =  Ph/s  x  Pk/h.)  The  resulting  hit  rate  can  also 
be  compared  to  existing  data  to  obtain  a  historical  perspective. 

5.4.5  Calculate  Fraction  of  the  Fleet  Lost  (FFL).  Using  the  survivor  rule,  Ph/s,  and  Pk/h,  the  FFL 
can  be  calculated.  Calculate  the  FFL  for  the  kill  levels  of  interest.  (FFL  =  1  -  [1  -  Pk/h  x  Ph/s]  **  Srt, 
where  Srt  equals  the  number  of  combat  sorties  per  aircraft.) 

5.4.6  Calculate  DVC.  From  the  FFL,  the  cost  of  lost  and/or  damaged  vehicles  can  be  calculated. 
In  order  to  calculate  the  total  cost  of  the  vehicles,  an  estimate  of  the  cost  of  returning  damaged  systems, 
deemed  to  be  unrepairable  and  lost  to  the  inventory,  should  be  included.  The  DVC  is  determined  by 
multiplying  the  percent  of  the  fleet  lost  by  the  number  of  aircraft  in  the  fleet  by  the  cost  of  each  aircraft 
lost.  (DVC  =  FFL  X  #A/C  x  A/C  production  cost.) 

5.4.7  Examine  the  Benefit  of  LFT.  To  estimate  the  potential  benefit  of  LFT  due  to  reduced  attrition, 
a  flaw  is  assumed  to  have  been  overlooked  in  the  design  and/or  vulnerability  analysis  of  the  aircraft.  This 
results  in  the  true  Pk/h  of  the  aircraft  actually  being  higher  than  the  assessed  spec  value.  It  is  further 
assumed  that  this  true  Pk/h  is  discovered  during  the  LFT  program  and  lowered  back  to  the  spec  value  (or 
some  determined  intermediate  value)  through  subsequent  redesign  (and  retrofit,  if  necessary).  By  repeating 
the  calculations  described  in  sections  5.4.5  and  5.4.6  (holding  probability  of  hit  constant),  to  obtain  a  DVC 
for  the  true  Pk/h,  and  comparing  with  the  original  estimate,  a  cost  savings  due  to  LFT  can  be  assessed. 

5.4.8  Estimate  Cost  of  LFT.  Estimate  the  cost  of  LFT  for  the  options  being  investigated.  The 
estimate  should  include  all  the  costs  associated  with  the  testing  including:  the  total  cost  of  performing  the 
tests,  special  facility  requirements,  and  the  cost  of  the  test  articles  required  (replicas,  components, 
subassemblies,  assemblies,  and  engineering  manufacturing  development  [EMD]  or  low-rate  initial 
production  [LRIP]  or  production  systems),  etc.  Because  the  relationship  between  cost  savings  (reduced 
attrition)  and  tire  total  investment  from  the  LFT  process  will  be  compared,  an  estimate  of  redesign  costs 
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and  any  retrofit  costs  should  also  be  included.  Since  the  cost  of  the  test  hardware,  possible  redesign,  and 
possible  retrofit  can  be  significant,  close  examination  of  the  actual  system  acquisition  circumstances 
(availability  of  hardware,  timing  with  regards  to  preliminary  design  review  [PDR]  and  critical  design 
review  [CDR],  likelihood  of  retrofit)  is  required  for  realistic  results.  The  ability  to  find  and  correct  the 
hidden  flaw,  within  the  scope  of  the  LFT  options  being  studied,  must  also  be  realistic. 

5.4.9  Compare  Attrition  Savings  With  Cost  of  LFT.  The  estimated  savings  in  attrition,  from 
section  5.4.7,  can  then  be  compared  to  the  estimated  cost  of  the  LFT  program  as  determined  in 
section  5.4.8.  The  total  dollar  savings  can  be  compared  with  the  LFT  investment  and,  if  desired,  the 
return  on  investment  can  be  easily  calculated.  (Consideration  should  be  given  to  constant  year  dollars.) 

5.4.10  Optimization.  Using  the  steps  previously  described  (sections  5.4.1-5.4.9),  an  optimum  strategy 
for  designing  an  LFT  program  can  be  developed.  The  most  cost-effective  strategy  for  LFT  can  be 
ascertained  by  combining  the  estimated  cost  of  the  various  test  options  with  the  predicted  attrition  cost 
related  to  the  options.  Prudent  combinations  of  LFT  options  including  component,  subsystem, 
subassembly,  assembly,  and  system-level  testing  should  be  evaluated. 

5.4.11  Example.  The  following  is  a  simple  example  using  the  procedure  on  a  hypothetical 
fighter-type  aircraft.  ITie  input  estimates  were  selected  only  to  demonstrate  the  methodology  and  are  not 
reflective  of  any  current  or  plaimed  system. 

Step  1;  Select  a  representative  sortie/flu'eat  scenario: 

Total  number  of  combat  sorties:  20,000 
Total  number  of  aircraft:  200 

"Allowable"  attrition  rate  (Pk/s):  .002  (for  K,  A,  and  B  kUls) 

Step  2:  Determine  production  rate  and  cost  of  system: 

Production  rate  (aiicraft/year):  50/yr  for  4  yr 
Production  cost  (per  aircraft):  $40M 

Step  3:  Determine  baseline  aircraft  vulnerability: 

Pk/h  (from  hypothetical  aircraft  specification) 

Pk/h  =  .3  (for  K,  A,  &  B  kills) 
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Step  4:  Calculate  estimated  hit  rate  (Ph/s): 

Pk/s  =  Ph/s  X  Pk/h, 
then  Ph/s  =  .0067 

Step  5:  Calculate  JhhL: 

From  survivor  rule: 

FFL  =  1  -  [1  -  Pk/h  X  Ph/s]  **  Srt 
Sit  is  number  of  sorties  per  aircraft:  100 
FFL  =  .18 

Step  6:  Calculate  DVC: 

DVC  =  FFL  X  #A/C  x  A/C  production  cost 
DVC  =  $1,440M 

Step  7:  Examine  benefit  of  LFT: 

Assume  true  vulnerability: 

Say,  Pk/h  actually  .4,  discovered  and  corrected  with  LFT 
DVC  (Pk/h  =  .4)  =  $1,920M 
DVC  (Pk/h  =  .3)  =  $1,440M 
Therefore,  savings  =  $480M 

Step  8:  Estimate  cost  of  LFT: 

For  this  example: 

Option  1  (assembly-level  testing  only)  =  $40M 

Option  2  (system-level  testing  complete  aircraft,  with  redesign  and  retrofit  costs  included) 
=  $200M 

Step  9:  Compare  attrition  savings  with  cost  of  LFT: 

If  the  system’s  Pk/h  can  be  lowered  from  .4  to  .3  using 
Option  1,  savings  =  $480M  less  $40M  investment  =  $440M 
Return  on  investment  =12:1 

Option  2,  savings  =  $480M  less  $200M  investment  =  $280M 
Return  on  investment  =  2.4:1 
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Step  10:  Optimization: 

For  this  example,  the  optimum  approach  is  apparent;  however,  more  complex  evaluations, 
such  as  inclusion  of  component  and  subassembly  testing,  redesign  without  retrofit,  etc.,  can 
be  made  using  this  approach. 

Option  1,  Total  costs  =  $1,440  (attrition)  plus  $40M  (LFT)  =  $1,480M 
Option  2,  Total  costs  =  $1,440  (attrition)  plus  $200M  (LFT)  =  $1,640M 

Conclusion:  For  this  simple  example.  Option  1  is  the  most  cost-effective  approach. 

5 .4. 12  Summary  Aircraft  Attrition-Based  Methodology.  In  summary,  the  attrition-based  approach  has 
significant  merit  for  evaluating  cost  vs.  payoff  for  LFT  of  aircraft.  The  methodology  allows  for  detail 
evaluation  of  the  various  LFT  options;  however,  care  must  be  exercised  in  choosing  realistic  inputs, 
particularly  with  regards  to  test  objectives  and  costs,  for  the  results  to  be  meaningful.  It  is  flexible, 
relatively  simple,  and  it  includes  (or  has  the  ability  to  include)  the  largest  and  most  important  costs  and 
benefits.  It  exploits  the  most  important  cause-effect  relationship  associated  with  LFT  (i.e.,  test  resources 
expended  result  in  reduced  attrition  and  saved  lives). 

5,5  Rayftsian  Methodology.  This  section  is  extracted  from  the  document  "Estimating  Expected  Value 
of  System  Testing,"  written  by  Stephen  M.  Robinson  from  the  University  of  Wisconsin-Madison  (Address: 
Department  of  Industrial  Engineering,  University  of  Wisconsin-Madison,  1513  University  Avenue, 
Madison,  WI  53706-1572;  email:  smr@cs.wisc.edu;  fax:  (608)  262-8454).  The  research  reported  here 
was  sponsored  by  the  U.S.  Army  Research  Office  under  Grants  DAAL03-92-G-0408  and  DAAH04-95-1- 
0149,  and  by  the  U.S.  Army  Space  and  Strategic  Defense  Command  under  Contract  DASG60-91-C-0144. 

5.5.1  Purpose  and  Scope.  This  paper  proposes  a  method  for  use  in  estimating  the  value  of  LFT  of 
systems  or  subsystems.  It  states  the  assumptions  under  which  the  method  is  proposed  for  use,  explains 
the  procedure  and  demonstrates  it  on  a  small  example,  and  finally  considers  possible  objections  and 
responses  to  these.  The  method  is  intended  for  use  in  supporting  a  decision  on  whether  to  request  a 
waiver  by  the  Secretary  of  Defense  of  LFT  for  a  particular  system  on  the  grounds  that  LFT  is 
"unreasonably  expensive  and  impractical,"  under  the  provisions  of  the  Live  Fire  Test  Law  (Title  10, 
Chapter  139,  Section  2366,  U.S.  Code). 
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The  effect  of  the  proposed  procedure  is  to  estimate  the  expected  value  of  the  information  gained  from 
the  test,  which  may  help  the  decision  maker  judge  whether  the  expense  of  LFT  is  or  is  not  "unreasonable." 
Of  course,  the  decision  as  to  how  large  a  cost  increment  is  "unreasonable"  is  a  matter  of  judgment,  and 
quantitative  analysis  methodology  cannot  replace  it.  However,  such  methodology  can  often  clarify  the 
elements  of  such  a  decision,  and  the  process  suggested  here  is  intended  to  do  so. 

The  method  proposed  is  based  on  well-known  and  widely  accepted  principles  for  measuring  the  value 
of  information.  For  example,  outlines  may  be  found  in  Raiffa  (1978)  (section  7.8)  and  Lindley  (1985) 
(section  7.9),  and  an  expository  treatment  with  several  worked  examples  is  in  Jackson  (1979).  However, 
the  particular  procedure  suggested  here  in  coimection  with  system  testing  has  not,  as  far  as  the  author 
knows,  been  implemented  for  any  real  systems,  and  the  example  provided  is  entirely  fictitious.  Therefore, 
this  paper  should  be  considered  as  a  proposal  for  further  development  and  testing  of  the  procedure  on  real 
systems,  rather  than  as  a  recommendation  for  immediate  implementation. 

5.5.2  Assumptions.  The  proposed  method  depends  on  the  following  assumptions: 

(1)  Bayesian  statistical  methodology,  in  particular  the  use  of  preposterior  analysis,  is  acceptable  for 
use  in  support  of  the  LFT  waiver  process. 

(2)  Costs  of  a  projected  LFT  can  be  estimated.  In  addition,  costs  of  (1)  withdrawing  the  system  from 
the  normal  development  process  (after  testing,  but  before  deployment)  for  redesign,  and 
(2)  withdrawing  the  system  from  the  field  after  deployment  (based  on  failure  after  deployment, 
possibly  accompanied  by  loss  of  life  or  injury  and  necessitating  recall  and/or  extensive 
modification  in  place)  can  be  estimated. 

(3)  The  effectiveness  of  the  proposed  test  can  be  estimated,  in  the  form  of  probability  distributions 
for  system  behavior  under  test  conditioned  on  (unobservable)  system  characteristics,  and  the 
decision  to  deploy  or  not  to  deploy  the  system  depends  only  on  those  characteristics. 

(4)  Expected  costs  provide  a  satisfactory  measure  for  decision-making  purposes. 

For  an  organization  of  limited  resources  facing  a  potentially  extremely  costly  decision,  assumption  4 
might  not  be  appropriate;  however,  for  governmental  bodies,  it  is  generally  reasonable.  It  is  important 
here  because  if  it  did  not  hold,  then  it  would  be  necessary  to  do  the  cost  calculations  with  utilities  instead 
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of  monetary  costs.  This  would  increase  the  difficulty  of  applying  the  model,  primarily  because  of  practical 
obstacles  to  assessing  a  utility  function  in  the  environment  under  consideration. 

5.5.3  Measuring  Value  Added  by  Testing.  In  this  section,  we  describe  the  proposed  procedure.  The 
description  is  brief  and  technical,  intended  only  to  make  precise  just  what  is  being  proposed.  The  next 
section  will  present  a  simple  example  to  illustrate  the  application  of  the  method. 

To  fix  the  notation,  consider  a  state  space  5,  points  of  which  specify  the  underlying  (unobservable) 
system  characteristics.  These  are  the  things  we  would  really  like  to  know  about  the  system  but  cannot. 
In  principle,  5  could  be  multidimensional  and  either  continuous  or  discrete.  In  practice,  because  of 
complications  in  estimating  the  required  data,  S  should  be  as  simple  as  possible.  In  the  example,  5  is  a 
discrete  space  consisting  of  16  points. 

We  shall  refer  in  what  follows  to  the  critical  region:  this  is  simply  the  subset  of  S  consisting  of  those 
states  s  e  S  that  would  result  in  a  decision  to  withdraw  the  system  from  service  after  deployment.  Thus, 
if  the  system’s  state  lies  in  the  critical  region,  then  the  system  is  to  be  considered  imfit  for  deployment. 

The  proposed  model  would  accommodate  multiple  critical  regions,  each  associated  with  a  different 
decision;  however,  we  consider  here  only  the  simplest  case  in  which  the  system  is  either  fit  for 
deployment  or  not. 

While  5  represents  the  information  about  the  system  that  we  would  like  to  know  but  cannot  observe, 
another  state  space  T  represents  test  outcomes.  For  example,  one  element  of  T  might  represent  a  certain 
combination  of  degrees  of  damage  incurred  in  a  specified  sequence  of  trials  using  specified  munitions. 
We  assume  that  the  space  T  represents  all  of  the  possible  outcomes  of  the  proposed  test.  Again,  this  space 
could  be  continuous  or  discrete,  and  its  points  could  be  of  any  dimension  desired.  However,  in  our 
example,  again  for  simplicity,  we  make  T  a  discrete  space  and  in  fact  put  its  points  in  one-to-one 
correspondence  with  those  of  S. 

We  assume  that  we  have  a  prior  probability  distribution  P^(s)  for  S.  This  distribution  may  be 
continuous  or  discrete.  It  may  also  be  diffuse  (uniform),  indicating  complete  ignorance  of  the  system’s 
state,  or  it  may  include  any  advance  information  that  we  have,  for  example  as  a  result  of  analysis  and 
modeling  or  of  previous  experimentation. 
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Second,  we  assume  that  we  have  a  conditional  probability  distribution  Pj|^rl5^  for  test  outcomes  t 
given  the  state  s.  It  is  worth  noting  that  the  specification  of  this  distribution  becomes  progressively  more 
burdensome  as  the  sizes  of  S  and  T,  and  the  fineness  of  detail  represented  in  them,  increase.  For  example, 
if  we  use  discrete  distributions  with  the  number  of  elements  in  S  being  o  and  the  number  in  T  being  t, 
then  the  number  of  conditional  probabilities  involved  is  ox.  This  number  becomes  less  manageable  as 
o  and  X  increase.  Therefore,  especially  when  discrete  distributions  are  used,  it  seems  advisable  to  use  the 
proposed  method  to  gather  information  about  only  the  most  important  aspects  of  the  system,  rather  than 
as  a  comprehensive  modeling  tool. 

The  final  decision  to  be  made  about  the  system  is  binary:  withdraw  the  system  from  the  acquisition 
process  (for  redesign  or  possible  cancellation)  or  else  proceed  through  the  acquisition  process.  With  the 
first  option,  we  assume  there  is  a  specified  cost  increment  note  that  this  cost  may  depend  on  s,  so 
that  we  may  face  different  kinds  of  costs  depending  on  the  state  of  the  system. 

If  we  choose  the  second  option  and  s  does  not  lie  in  the  critical  region,  then  there  is  no  cost 
increment,  since  the  process  is  to  proceed  normally.  On  the  other  hand,  if  we  choose  the  second  option 
and  S  does  lie  in  the  critical  region,  then  there  is  a  "disaster"  cost  this  is  understood  to  include  both 
the  costs  of  withdrawing  the  system  from  the  field  after  deployment,  or  of  making  emergency 
modifications  to  it  in  place,  and  any  quantification  that  we  may  wish  to  make  of  the  intangible  factors 
associated  with  system  nonperformance  (casualties,  readiness  shortfalls,  bad  publicity,  and  so  on).  In 
addition  to  Cj)(s),  we  must  also  pay,  in  this  case,  the  rework  cost  Cg(s). 

Recall  that  the  true  system  state  is  unobservable;  therefore,  our  decision  has  to  be  made  under 
xmcertainty.  Suppose  that  at  the  time  of  decision  we  have  a  probability  distribution  p(s)  over  system 
states;  this  could  be  either  the  prior  distribution  P^s)  already  mentioned,  or  another  distribution  based  on 
additional  information  (for  example,  information  from  testing).  In  order  to  fix  notation,  in  the  rest  of  this 
paper,  we  assume  that  we  are  using  discrete  distributions,  and  we  use  sums;  the  changes  needed  if  the 
distributions  were  continuous  would  be  to  replace  the  sums  by  integrals. 

We  make  the  decision  to  deploy  or  not  to  deploy  based  on  minimizing  total  expected  cost.  If  we 
decide  not  to  deploy,  this  expected  cost  will  just  be  the  expected  rework  cost,  namely 

EC„(p)  =  C^{s)p{s).  (1) 
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On  the  other  hand,  if  we  deploy  the  system  and  s  does  not  belong  to  the  critical  region  CR,  then  we  pay 
nothing,  whereas,  if  we  deploy,  and  s  does  belong  to  the  critical  region,  then  we  pay  Qjfsl  +  Cj^(s). 
Therefore,  the  expected  cost  associated  with  deployment  is 


ECi(p)  = 

sgCR 


Cj)(s)  +  C^(s)]p(5). 


(2) 


The  total  expected  cost  associated  with  our  decision,  given  the  probability  distribution  p(s),  is  then 

EC(p)  :=  min  {eCq(p) ,  EC^ (p)}  .  (3) 

For  later  reference,  we  note  that  each  of  jEC^-and  ECj  is  linear  in  the  probabilities  p(s);  therefore,  the 
expected  cost  EC  is  a  concave  function  of  these  probabilities. 

With  these  concepts  in  place,  we  can  specify  the  procedure  that  we  propose.  The  basic  idea  is 
extremely  simple:  to  compare  the  expected  system  costs  that  we  face  with  and  without  LFT.  The  excess 
of  the  costs  without  LFT  over  the  costs  with  LFT  represents  the  expected  value  of  the  information  to  be 
gained  from  die  test.  The  person  responsible  for  requesting  the  waiver  can  then  use  this  expected  value, 
together  with  the  predicted  cost  of  carrying  out  the  test  itself  (which  is  not  included  in  these  calculations), 
to  support  a  decision  on  whether  to  proceed  with  testing  or  to  request  a  waiver  on  the  grounds  that  LFT 
is  "unreasonably  expensive  and  impractical." 

To  make  the  calculation,  we  first  calculate  the  prior  probability  distribution  of  the  test  of  the  outcomes 
by 


Pt(^)  =  Y,  PT\S^t\s)Ps(s), 

s 

and  then  the  conditional  probability  distribution  of  states  given  outcomes  using 
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If  we  decide  not  to  perform  the  test,  our  expected  cost  is  given  by  EC{P^),  defined  by  (3)  with 
the  probabilities  p(s)  being  replaced  by  P^s).  On  the  other  hand,  if  we  do  test  and  we  observe  outcome 
t,  then  our  expected  system  cost  is  given  by  EC(Pg^f).  The  prior  probability  of  observing  t  is  Pj(t),  so 
our  total  expected  system  cost  given  a  decision  to  test  is 


Cj-  =  X)  EC{Ps\,)Pj.(t). 

t 


Note  that  since  we  assume  the  test  outcomes  t  to  be  disjoint  and  exhaustive,  we  have 

=  E  =  E  [P(st)/PT(t)]PT(.t)  =  E^s|/(^)^r(^)> 

t  t  t 


where  st  indicates  the  event  "s  and  r."  The  probabilities  Pj{t)  sum  to  1,  so  the  concavity  of  the  expected 
cost  EC  as  a  function  of  the  probabilities  means  that  Cn>  Cj  that  is,  if  we  test,  then  our  expected 
system  cost  will  not  be  greater  than  if  we  do  not  test.  The  savings  from  testing,  namely  can 

then  be  compared  to  the  cost  of  testing  to  help  support  a  decision  on  whether  to  test  or  to  seek  a  waiver. 

5.5.4  Illustrative  Example.  In  this  section,  we  provide  an  example  to  illustrate  application  of  the 
proposed  method.  This  example  is  a  much  simplified  version  of  a  decision  problem  involving  an  aircraft. 
Our  goal  is  to  estimate  the  value  of  LPT  for  this  system. 

In  the  example,  we  consider  two  types  of  munitions,  rocket  (R)  and  small-arms  round  (S),  that  might 
be  employed  against  the  system.  If  a  hit  occurs,  we  suppose  the  damage  can  be  expressed  in  terms  of  one 
of  four  categories:  catastrophic  kill  (K),  sufficient  damage  to  cause  mission  abort  (A),  damage  requiring 
later  maintenance  but  permitting  mission  completion  (C),  and  negligible  damage  (N).  Of  course,  a  more 
detailed  example  could  subdivide  the  aircraft  into  numerous  regions,  some  more  important  than  others, 
and  could  consider  the  effects  of  multiple  hits  in  one  or  more  of  these  regions. 

In  terms  of  the  previous  notation,  our  state  space  S  has  16  elements,  each  consisting  of  two  letters 
selected  from  the  set  {K,  A,  C,  N}:  for  example,  the  pair  KC  indicates  that  if  the  aircraft  is  hit  by  a  rocket 
we  can  expect  catastrophic  kill,  but  if  it  is  hit  by  a  small-arms  rovmd,  we  can  expect  a  level  of  damage 
permitting  mission  completion. 
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Table  22  gives  our  prior  (pretest)  probability  of  each  state  in  S.  We  assume  the  degrees  of  damage 
caused  by  the  two  different  types  of  munitions  are  independent,  so,  for  example,  the  probability  of  the  pair 
KC  just  mentioned  would  be  (.20)  (.35)  =  .07.  This  independence  assumption  is  not  necessary. 


Table  22.  Prior  Probabilities  of  System  States 


Munition 

K 

A 

C 

N 

Rocket 

.20 

.05 

Small- Arms  Round 

.05 

.10 

.35 

.50 

We  shall  take  the  critical  region  to  consist  of  those  states  that  (1)  include  a  catastrophic  kill  by  either 
munition,  or  (2)  include  a  "mission  abort"  event  from  small-arms  fire.  Therefore  the  noncritical  region 
consists  of  all  those  pairs  whose  first  letter  is  A,  C,  or  N,  and  whose  second  is  C  or  N.  Simple 
computations  using  the  data  in  Table  22  show  that  the  prior  probability  of  the  systems  being  in  the  critical 
region  is  .32.  The  choice  of  critical  and  noncritical  regions  is  completely  at  the  disposal  of  the  modeler, 
and  we  could  have  taken  any  other  subset  of  S  to  be  the  critical  region. 

Table  23  gives  the  conditional  probabilities  of  test  outcomes  t,  given  true  system  states  s.  The 
outcome  space  T  actually  consists  of  16  points,  the  notation  for  which  is  identical  to  that  for  the  elements 
of  S.  Therefore,  the  full  table  of  conditional  probabilities  has  16  x  16  =  256  entries.  However,  to  reduce 
the  requirements  for  display  of  data,  we  make  several  assumptions,  none  of  which  is  necessary  to  the 
analysis.  First,  we  assume  that  for  each  s,  s',  t,  and  t',  the  events  "for  the  rocket,  the  true  state  is  s  and 
the  test  outcome  is  t"  and  "for  the  small-arms  round,  the  true  state  is  s'  and  the  test  outcome  is  t'  "  are 
independent;  this  assumption  implies  independence  of  t  and  t',  as  well  as  our  earlier  assumption  of 

independence  of  s  and  s',  and  under  it  the  probability  P(ff'|5j' )  can  be  computed  as  P(t|5)P(t'|s')- 
Second,  we  assume  that  the  conditional  probability  of  outcome  given  state  is  the  same  for  the  two  types 
of  mimitions.  Under  these  assumptions,  the  full  table  of  256  probabilities  can  be  recovered  from  the 
following  4x4  table,  in  which  true  system  states  are  shown  along  the  top,  and  test  outcomes  along  the 
side. 
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Table  23.  Probability  of  Test  Outcome  Given  System  State 


Outcome 

(below) 

K 

A 

C 

N 

K 

.9 

.1 

0 

A 

.1 

.8 

.1 

0 

C 

.1 

.8 

.1 

N 

0 

0 

.1 

.9 

This  table  indicates  that  we  have  considerable  confidence  in  the  discriminatory  power  of  our  test.  Less 
optimism  about  the  test  would  have  resulted  in  a  more  diffuse  set  of  conditional  distributions. 

From  data  in  Tables  22  and  23,  we  calculate  the  prior  probabilities  of  the  test  outcomes  from  the 
relationship 

P  2-(f)  =  ^  P  jyj(/ls)P  gis). 

s 

Because  of  the  independence  assumption,  instead  of  displaying  a  table  of  16  items,  we  can  display 
in  Table  24  two  rows  of  4  items  each,  corresponding  to  the  rocket  and  the  small-arms  round;  the 
probability  of  a  composite  outcome  is  then  found  by  multiplication,  just  as  it  was  with  the  prior 
probabilities  for  system  state. 


Table  24.  Prior  Probabilities  of  Test  Outcomes 


Munition 

K 

A 

C 

N 

Rocket 

.215 

.340 

.360 

.085 

Small-Arms  Round 

.055 

.120 

.340 

.485 
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Next  we  calculate  the  conditional  probabilities  of  true  system  states  s  given  test  outcomes  t,  using  the 
formula 

and  using  the  data  from  Tables  22,  23,  and  24.  Again,  because  of  independence,  we  can  reduce  the 
amount  of  displayed  data  to  two  tables,  one  for  the  rocket  and  one  for  the  small-arms  round.  These  are 
shown  as  Tables  25  and  26.  TTie  true  system  states  are  at  the  side,  and  the  test  outcomes  are  along  the 
top.  Numbers  are  rounded  where  necessary  to  make  the  probabilities  in  a  distribution  sum  to  1. 


Table  25.  Probability  of  True  System  State  Given  Test  Outcome 
(Rocket) 


True  State 
(below) 

K 

A 

C 

N 

K 

.8372 

0 

A 

.1628 

.8235 

0 

C 

0 

.1177 

.8889 

.4706 

N 

0 

0 

.0139 

.5294 

Table  26.  Probability  of  True  System  State  Given  Test  Outcome 
(Small-Arms  Round) 


True  State 
(below) 

K 

A 

C 

N 

K 

.8182 

0 

A 

.1818 

.6666 

.0294 

0 

C 

0 

.2917 

.8235 

.0722 

N 

0 

0 

.1471 

.9278 
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Now  let  us  suppose  for  simplicity  that  the  rework  cost  Cg(s)  is  $2B  (billion)  for  each  5,  and  the 
disaster  cost  Cjy(s)  is  $5B  for  each  state  5  that  is  in  the  critical  region  CR  (it  is  zero  for  all  other  states). 
If  we  are  given  some  probability  distribution  p(s)  over  system  states,  let  us  define 

pCR  =  Y. 

seCR 


this  is  simply  the  probability  that  the  system’s  true  state  is  in  the  critical  region.  With  the  rework  and 
disaster  costs  just  given,  the  calculations  in  (l)-(3)  become  relatively  simple.  In  particular,  if  we  decide 
not  to  deploy,  we  just  pay  $2B,  so  ECq(p)  =  2.  If  we  deploy  and  the  system  is  good  (its  state  s  is  not  in 
CR),  then  we  pay  nothing,  whereas,  if  we  deploy  and  seCR,  then  we  pay  $7B.  Therefore, 
ECj(p)  =  Ipcg,  and  we  have 


EC(p)  =  min{2,7pcr}. 

We  shall  calculate  and  Cj-  by  replacing  the  distribution  p(s)  by  P^s)  and  /*5|7’(j  |0  respectively. 

If  we  do  not  test,  there  is  a  probability  of  .32  that  the  system  state  is  in  the  critical  region,  so  the 
expected  cost  will  be 


=  min<2,7(.32)>  =  min  {2,2.24}  =  2, 
and  we  will  decide  not  to  deploy  the  system. 

In  order  to  determine  the  expected  cost  Cj-  if  we  decide  to  test,  we  first  calculate  for  each  test  outcome 
t  the  probability  that  the  true  system  state  5  belongs  to  the  noncritical  region  as  defined  previously.  (We 
are  actually  going  to  use  the  probability  that  it  is  in  the  critical  region,  but  the  complementary  noncritical 
probability  is  simpler  to  calculate  in  this  case.)  For  example,  suppose  the  outcome  state  is  AC.  Thai  from 
Table  25  the  probability  that  the  state  with  respect  to  rocket  damage  is  A,  C,  or  N  (given  rocket  test 
outcome  A)  is  .8235  +  ,1177  +  0  =  .9412,  From  Table  26,  the  probability  that  die  state  with  respect  to 
small-arms  damage  is  C  or  N  (given  small-arms  test  outcome  Q  is  .8235  +  .1471  =  .9706.  Therefore,  the 
probability  that  we  are  in  the  noncritical  region,  given  test  outcome  AC,  is  (.9412)  (.9706)  =  .9135.  These 
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probabilities  are  summarized  in  Table  27.  The  outcomes  for  the  rocket  are  along  the  side,  while  those  for 
the  small-arms  round  are  along  the  top. 


Table  27.  Probability  That  System  Is  in  Noncritical  Region  Given 
Joint  Test  Outcome 


Rocket 

(below) 

K 

A 

C 

K 

0 

.0475 

.1628 

> 

0 

.2745 

.9135 

.9412 

C 

0 

1 

N 

0 

.2917 

.9706 

1 

If  we  do  test,  then,  for  example,  consider  the  possibility  that  we  observe  test  outcome  KA.  Referring 
to  Table  27,  we  see  there  is  a  probability  of  .0475  that  the  system  is  not  in  the  critical  region;  hence = 
1  -  .0475  =  .9525,  so  the  expected  cost  if  we  deploy  the  system  is  7  (.9525)  =  6.6675,  while  the  expected 
cost  without  deployment  is  2.  Accordingly,  we  will  not  deploy  the  system,  and  our  expected  cost  in  case 
we  observe  KA  is  2.  By  making  this  computation  for  each  possible  outcome,  we  can  see  that  we  will 
decide  to  deploy  the  system  in  6  of  the  16  possible  test  outcomes:  namely,  those  for  which  t  is  given  by 
a  pair  XT,  with  X  being  A,  C,  or  N  and  Y  being  C  or  N.  By  referring  to  Table  24,  we  can  calculate  the 
probability  of  the  event  E  consisting  of  observing  1  of  these  6  outcomes  to  be  .6476,  so  that  the 
probability  of  observing  1  of  the  other  10  outcomes  is  .3524.  In  the  latter  case,  we  will  not  deploy  the 
system,  and  we  will  pay  the  rework  cost  of  $2B.  In  the  former  case,  we  will  deploy  the  system,  and,  as 
we  have  seen,  our  expected  cost  will  be  IpcR- 

For  each  of  the  six  test  outcomes  we  can  calculate  the  probability  that  the  true  system  state  lies 
in  the  critical  region  by  referring  to  Table  27  and  taking  the  complementary  probabilities  for  those 
outcomes  in  which  we  are  interested.  We  display  those  in  Table  28,  along  with  tire  probabilities  Pj(t)  for 
each  of  the  six  outcomes. 
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Table  28.  Probability  That  System  State  Is  in  Critical  Region  Given 
Selected  Outcomes 


Outcome  t 

P-jit) 

P(s  e  CR\t) 

AC 

.1156 

.0865 

CC 

.1224 

.0294 

NC 

.0289 

.0294 

AN 

.1649 

.0588 

CN 

.1746 

0 

NN 

.0412 

0 

For  each  of  the  six  test  outcomes  t  listed  in  Table  28,  the  conditional  expected  cost  of  the  optimal 
decision  (deploy),  given  fliat  we  have  obsCTved  t,  is  lP{s  e  CR 1 1) .  By  multiplying  this  conditional  cost 
by  the  probability  P j(t),  shown  in  the  first  column  of  the  table,  and  adding,  we  find  that  the  component 
of  total  expected  cost  attributable  to  observing  one  of  the  these  six  outcomes  is  .1690.  We  have 
previously  seen  that  the  conditional  expected  cost  in  each  of  the  other  10  outcomes  is  2,  and  the 
probability  that  we  will  observe  1  of  those  10  outcomes  is  .3524.  Therefore  the  component  of  total 
expected  cost  attributable  to  observing  1  of  those  10  outcomes  is  (2)  (.3524)  =  .7048.  As  these  two 
groups  exhaust  the  set  of  possible  outcomes,  the  total  expected  cost  given  a  decision  to  test  is 

Cj  =  .1690  +  .7048  =  .8738 . 


Therefore,  provided  the  cost  of  the  test  is  less  than  the  difference  C^f  -  Cj-of  $1.1262B,  we  would  expect 
to  gain  by  a  decision  to  test. 

5.5.5  Possible  Objections  and  Responses.  Here  we  consider  some  possible  reasons  that  might  be 
advanced  against  this  procedure  and  provide  some  responses  to  these. 

(1)  The  procedure  is  complicated,  and  it  requires  a  lot  of  numbers.  This  is  true.  However,  going 
through  the  process  of  making  the  cost  calculations  may  cause  us  to  think  more  about  the  process 
and  to  get  better  data  than  we  otherwise  would.  In  the  process,  we  may  come  out  with  a  better 
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decision  that  we  would  otherwise  have  made.  In  addition,  the  procedure  is  quite  flexible,  in  that 
the  system  states  and  outcomes  considered  may  be  extremely  coarse  (perhaps  only  two  states, 
serviceable  or  unserviceable)  or  quite  detailed,  and  the  work  may  be  adjusted  accordingly. 

(2)  We  may  have  to  guess  at  a  lot  of  the  data.  This  is  also  true.  However,  the  kind  of  data  required 
here  will  be  required  for  any  rational  decision  to  test  or  not.  Some  of  it  may  be  estimated  from 
previous  tests  of  the  system  under  consideration,  or  of  other  similar  systems,  or  even  from  such 
devices  as  combat  simulations.  The  difference  in  a  "judgmental"  decision  is  that  the  places  where 
one  guesses  at  data  are  not  visible.  An  attempt  to  set  out  an  orderly  calculation,  as  is  suggested 
here,  may  result  in  a  better-documented  case  for  whatever  action  is  chosen. 

(3)  We  should  be  using  utilities  and  not  expected  monetary  value  (EMV).  This  issue  was  mentioned 
at  the  beginning.  The  contention  here  is  that  EMV  is  a  reasonable  criterion  for  governmental 
decisions,  at  least  about  issues  that  are  unlikely  to  bankrupt  the  coimtry.  System  tests  of  the  kind 
we  are  considering  here  should  fit  within  this  framework. 

5.5.6  Summary.  We  have  presented  a  method  for  estimating  the  EMV  contributed  by  a  decision  to 
test  a  system  and  have  illustrated  its  use  on  a  simple  numerical  example.  We  have  suggested  that  a 
systematic  decision  procedure  like  the  one  outlined  here  may  be  useful  in  supporting  a  decision  on 
whether  to  seek  a  waiver  of  LET,  since  the  information  accumulated  during  the  decision  process  helps  to 
establish  whether  LET  would  be  "unreasonably  expensive  and  impractical." 

6.  RECOMMENDATIONS  AND  CONCLUSIONS 

This  report  focuses  on  the  four  areas  outlined  in  the  Statement  of  Work  for  this  study.  We  have 
detailed  the  V/L  process  structure  as  a  framework  for  describing  a  simulated  target-threat  encounter.  This 
taxonomy  was  developed  when  the  sunharities  between  different  ballistic  threats  (indirect-fire  munitions, 
direct-fire  munitions,  HEI  rounds)  against  various  targets  (logistical  targets,  armored  vehicles,  aircraft) 
became  obvious.  Since  the  original  development,  it  has  been  understood  that  this  taxonomy  also  holds 
for  other  threat  mechanisms  -  electronic,  chemical,  and  biological  warfare  against  defense  materiel.  Using 
the  taxonomy  to  describe  a  specific  simulation  is  useful  in  showing  the  metrics  produced  at  each  level  and 
the  methodologies  used  in  the  mapping  functions. 
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Models  and  analysis  methods  that  are  to  be  applied  in  support  of  LFT  (PMS  II)  should  include 
information  at  all  stages  at  which  the  vulnerability  of  the  system  can  be  affected.  Thus,  for  each  threat 
damage  mechanism,  a  complete  set  of  vulnerability  information  includes  component  vulnerability  data, 
subsystem-component  interactions  (fault  trees),  and  subsystem  interactions.  Additionally,  if  damage 
mechanisms  are  not  independent,  combined  effects  for  components,  subsystems,  and  multiple  subsystems 
must  be  included.  For  whatever  type  of  model  is  applied,  it  is  reasonable  to  assume  an  imcertainty  for 
each  stage  of  input  as  well  as  an  uncertainty  each  time  the  model  aggregates  damage  among  components, 
within  subsystems,  and  between  subsystems  (to  the  system  level).  The  imcertainty  associated  with 
aggregation  is  due  to  possible  dependent  effects  among  various  components  and  subsystems  or  among 
various  damage  mechanisms  or  both.  Different  types  of  information  are  obtained  through  a  variety  of 
analyses  and  test  procedures,  including  simulation  modeling  and  nondestructive  testing  along  with 
component,  subsystem,  and  full-up  LFT.  The  tradeoffs  between  full-up  LFT  and  other  information- 
gathering  plans  must  address  the  ability  to  obtain  information  at  each  of  these  stages. 

While  agreeing  with  the  spirit  of  the  live-fiire  law,  we  realize  that  information  can  be  substituted  for 
testing.  Thus,  in  attempting  to  formalize  the  LFT  waiver  process,  we  have  outlined  a  set  of  four  risk- 
benefit  assessm^t  methodologies,  each  of  which  is  an  information-based  approach.  Each  one  attempts 
to  determine  whether  a  full-up  LFT  program  for  any  particular  system  is  "unreasonably  expensive  and 
impractical."  We  emphasize  that  any  formalized  methodology  should,  in  its  final  form,  address  all  types 
of  information  and,  furthermore,  that  this  information  should  be  of  sufficient  accuracy  and  at  a  sufficient 
level  of  confidence  such  that  undetected  vulnerabilities  would  be  very  unlikely  in  the  systems  produced. 
None  of  the  four  methodologies  is  fully  developed,  but  we  are  encouraged  by  the  fact  that  each  of  them 
shows  promise  in  contributing  to  the  overall  goal  of  a  formalized  procedure.  The  report  presents  these 
four  methodologies,  labeled  as  follows: 

•  Decision-Tree  Methodology. 

•  Knowledge-Based  Methodology. 

•  Attrition-Based  Methodology. 

•  Bayesian  Methodology. 
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The  decision-tree  methodology  incorporates  a  series  of  questions  pertaining  to  the  costs  and  benefits 
of  full-up  LFT  of  a  system.  It  represents  an  expert  system  of  sorts  in  that  the  answers  to  the  various 
questions  will  lead  the  user  on  a  path  that  will  eventually  provide  a  decision  concerning  whether  or  not 
to  apply  for  a  waiver  from  fuU-up  LFT  while  simultaneously  suggesting  alternative  testing  strategies.  This 
methodology  could  work  well,  combined  with  any  of  the  other  approaches,  to  act  as  a  filter  to  grant 
waivers  to  systems  that  are  clearly  unreasonably  expensive  and  impractical,  before  going  into  the  highly 
detailed  analyses  required  by  the  other  methodologies. 

TTie  knowledge-based  methodology  is  based  upon  the  following  principles: 

•  The  goal  of  analysis  is  to  provide  information  for  decision  makers. 

•  The  reason  for  testing  is  to  provide  data  that  is  needed  to  improve  analyses. 

•  There  are  many  data-gathering  techniques,  each  offering  benefits  in  quality  of  data  and  carrying  its 
own  price.  It  is  necessary  to  assemble  a  data-gathering  plan  that  uses  all  appropriate  techniques 
to  obtain  data  of  sufficient  quality  at  a  cost  that  is  low  relative  to  other  plans. 

In  this  methodology,  vulnerability  analysis  methodologies  currently  in  use  are  used  to  quantify  the 
quality  of  information. 

The  attrition-based  methodology  compares  estimated  savings  due  to  reduced  losses  with  the  estimated 
cost  of  executing  a  fixll-up  LFT  program.  In  the  methodology,  it  is  assumed  that  a  previously  undetected 
vulnerability  "flaw"  exists  in  the  system,  causing  the  system’s  true  vulnerability  to  be  higher  than 
originally  assessed.  By  detecting  the  flaw  through  fuU-up  LFT  and  fixing  it  through  redesign  (and  retrofit, 
if  necessary),  the  system  vulnerability  can  be  reduced.  By  comparing  "before"  and  "after"  vulnerabilities, 
an  estimated  savings  can  be  calculated  and  compared  to  the  total  cost  of  the  full-up  LFT,  redesign,  and 
retrofit. 

The  effect  of  the  Bayesian  methodology  is  to  estimate  the  expected  value  of  the  information  gained 
from  the  test.  While  using  probable  cost  (as  does  the  knowledge-based  methodology),  the  Bayesian 
methodology  also  employs  probable  gain  (from  an  experiment)  in  establishing  a  figure  for  risk-benefit 
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Within  this  report,  we  have  presented  examples  in  applying  two  of  the  methodologies  (attrition-based 
and  knowledge-based). 

All  the  risk-benefit  analysis  methods  to  LFT  are  still  in  the  early  stages  of  development.  Some  are 
more  complete  than  others;  but,  in  fact,  it  must  be  understood  that  the  development  and  application  of 
risk-benefit  analysis  methods  to  LFT  are  still  in  the  research  stage.  All  of  the  methods  presented  show 
sufficient  promise  to  be  included  in  this  document.  However,  each  one  has  shortcomings  to  be  overcome, 
problems  to  be  solved,  and  concepts  to  be  proven.  The  best  solution  may  prove  to  be  a  hybrid  of  the  best 
ideas  from  all  four,  the  decision-tree  methodology,  in  particular,  seems  to  be  compatible  with  each  of  the 
others  in  that  it  might  serve  as  a  prefilter  for  arriving  at  a  decision  concerning  LFT.  It  may  also  prove 
that  the  needs  of  the  services  are  sufficiaitly  different  and  that  more  than  one  standard  methodology  is 
required. 
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APPENDIX  A: 

QUANTIHCATION  OF  FUZZY  ACCURACY  AND  UNCERTAINTY 
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A.l.  "Probably"  and  "Surely." 


The  trickiest  part  of  this  procedure — ^whether  it  is  called  "Fuzzy  math"  or  not — ^is  understanding  the 
meaning  of  a  Fuzzy  datum. 

(The  reason  for  putting  "Fuzzy"  in  quotes  is  because  the  procedure  being  presented  here  uses  very 
little  of  the  large  body  of  work  that  has  evolved  under  the  title  "Fuzzy  mathematics."  The  procedure 
described  here  requires  just  enough  to  make  recognition  of  Fuzzy  mathematics  an  ethical  necessity.) 

In  the  following  discussion,  the  word  "datum"  (plural  "data")  will  refer  to  values  of  fundamental 

quantities— -determined  from  tests,  data  analyses,  or  wherever — ^that  are  used  as  inputs  to 

vulnerability  models.  The  term  "information"  wiU  refer  to  the  outputs  of  such  models. 

Vulnerability  information  is  subsequently  used,  in  conjimction  with  other  information,  by  decision¬ 
makers. 

The  problem  being  addressed  is  the  quantification  of  uncertainty  in  the  vulnerability  information  being 
given  to  decision-makers.  In  particular,  it  is  necessary  to  relate  imcertainty  in  fundamental  vulnerability 
data — as  used  for  inputs  to  vulnerability  analysis  models — to  the  outputs  of  those  models  that  form  the 
bulk  of  vulnerability  information  for  decisions. 

The  approach  taken  is  to  present  a  model  of  a  fundamental  input  datum,  which  quantitatively  includes 
the  uncertainty  present  in  the  datum.  This  imcertainty  is  then  propagated  through  the  vulnerability  analysis 
models,  arriving  at  a  quantitative  expression  for  the  uncertainty  in  the  outputs. 

The  model  of  an  uncertain  (Fuzzy)  input  datum  is  based  upon  the  following  postulates: 

1.  All  data  values  are  ultimately  decided  by  an  analyst. 

2.  Generally,  for  any  given  datum,  there  is  a  range  within  which  the  analyst  believes  the  datum  value 
"probably"  lies. 

3.  Generally,  an  analyst  can  state  an  upper  bound  above  which  he/she  believes  that  the  datum  value 
surely  doesn’t  lie.  Similarly,  a  lower  bound  can  be  stated. 
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NOTE  WELL:  Hiese  values  have  no  statistical  significance.  No  attempt  is  being  made  to  model  the 
distribution  of  a  random  variable.  Rather,  this  is  merely  an  attempt  to  model  an  analyst’s  estimation  of 
a  single  value. 

If  we  were  to  represent  these  concepts  graphically,  using  the  value  "1"  to  represent  a  belief  in  the 
appropriateness  of  associated  values  with  the  datum,  and  "0"  to  represent  a  belief  in  the  inappropriateness, 
a  model  of  an  analyst’s  estimate  of  the  value  of  a  datum  could  be  portrayed  as  in  Figure  A-1.  The 
purpose  of  the  line  connecting  the  four  points  is  merely  as  an  aid  to  the  eye. 

The  problem  lies  in  quantitatively  interpreting  the  terms  "probably"  and  "surely"  and  conveying  that 
interpretation  to  the  analysts  whose  data  and  associated  confidence  is  being  modeled. 


Figure  A-1.  Model  of  a  datum  value  estimate. 


112 


Recall  that  we  are  not  modeling  the  distribution  of  the  actual  values  of  the  datum;  rather,  we  are 
modeling  an  analyst’s  estimate  of  a  single  value.  Thus,  it  is  appropriate  to  consider  a  gedanken 
experiment  performed  on  analysts. 

Consider  a  group  of  analysts  competitively  estimating  the  values  of  a  list  of  data.  Each  analyst  is  to 
give  the  range  in  which  he/she  believes  each  datum  value  to  lie.  Afterwards,  the  data  are  evaluated.  For 
each  analyst,  the  widths  of  the  ranges  given  for  each  datum  are  summed;  and  the  winner  is  the  analyst 
with  the  smallest  sum.  However,  any  analyst  is  disqualified  who  fails  to  include  at  least  X  of  the  data 
values  within  the  ranges  assigned.  Therefore,  the  optimum  strategy  would  be  to  assign  ranges  to  the  data 
that  are  just  large  enough  that  the  analyst  is  correct  X  of  the  time. 

With  this  model  of  the  estimation  process,  we  can  put  a  quantitative  definition  on  terms  such  as 
"probably"  and  "surely."  For  example,  a  possible  definition  of  "probably"  is  "Let  X  =  50."  The  range 
specified  in  postulate  2  is  thus  the  smallest  range  such  that  the  analyst  expects  more  than  half  of  his  range 
estimates  to  include  the  actual  datum  value.  Similarly,  "surely"  could  mean  "large  (small)  ©trough  to  be 
correct  in  99  out  of  100  estimates."  In  fliis  paper,  again,  note:  We  are  not  attempting  to  determine  the 
distribution  of  any  variable.  Raflier,  we  are  attempting  to  calibrate  the  "margin  of  error"  that  an  analyst 
assigns  to  the  values  he/she  attaches  to  disparate  data. 

For  convenience,  we  will  generically  refer  to  the  ranges  in  which  values  are  believed  to  "probably" 
and  "surely"  lie  as  the  "Fuzziness"  of  a  datum.  A  datum  value  expressed  in  terms  of  such  ranges  will  be 
referred  to  as  a  "Fuzzy  number." 

A.2.  Reduction  in  Assigned  Uncertainty. 

Now,  consider  the  role  of  actual  knowledge  in  this  process.  Clearly,  an  analyst  must  have  some 
knowledge  that  is  pertinent  to  every  datum.  In  practice,  analysts  use  surrogate  information  or 
computations  of  varying  complexity  upon  which  to  base  estimates. 

The  availability  of  a  dependable  measurement  of  a  datum  allows  an  analyst  to  reduce  its  Fuzziness. 
This  is  propagated  through  the  analysis  into  a  reduction  in  the  Fuzziness  in  the  final  vulnerability 
information,  thus  implying  greater  certainty  in  the  result. 
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A.3.  Interpolation  Between  "Probably"  and  "Surely." 


Note  that,  thus  far,  only  four  values  have  been  discussed;  two  that  delineate  the  "probable"  region 
and  the  two  extrema.  In  terms  of  expected  correct  estimates,  these  were  construed  to  be  50%  and  99%. 
Recalling  that  these  values  are  subjective  regardless  of  the  prior  data  and  experience,  it  would  be  naive 
to  attempt  a  high-resolution  calibration  of  the  intervals  between  the  "probable"  range  and  each  of  the 
extrema.  However,  requests  for  other  than  50%  confidence  are  common.  Following  the  procedure 
outlined  previously,  it  would  be  possible  to  ask  an  analyst  to  increase  the  range  for  each  datum  estimate 
such  that  he/she  would  expect  to  be  correct  80%  of  the  time,  for  example.  However,  it  is  unrealistic  to 
place  the  burden  of  additional  intervals  upon  analysts  who  have  estimated  the  50%  and  99%  confidence 
intervals  for  data.  Therefore,  a  scheme  is  proposed  to  specify  X  intervals  from  the  50%  and  99%  intervals 
already  derived. 

We  make  two  observations: 

1.  An  increase  in  requested  confidence  should  always  require  an  increase  in  the  width  of  the 
associated  interval. 

2.  The  required  increase  in  interval  per  increase  in  confidence  becomes  larger  as  the  required 
confidence  approaches  1. 

A  well-known  function  that  displays  these  characteristics  is  the  cumulative  normal.  Having  no  further 
information,  we  propose  using  the  cumulative  normal  to  gauge  the  size  of  an  interval  corresponding  to 
a  confidence  between  50%  and  99%.  This  gauge  is  shown  in  Figure  A-2. 

NOTE:  It  must  be  emphasized  that  the  use  of  the  mathematical  function  commonly  referred  to 
as  the  "cumulative  normal"  is  strictly  an  artifice  proposed  by  the  author  which  exhibits  the  desired 
characteristics  listed  previously.  Although  the  cumulative  normal  function  finds  frequent 
application  as  a  probability  distribution,  no  probabilistic  implications  are  being  made  by  its  use 
here. 
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Figiire  A-2.  Model  of  a  "gauged"  datum  value  estimate, 

It  is  also  noted  that  a  linear  gauge,  as  implied  by  a  straight  line  connection  between  the  respective 
inner  and  outer  values  of  a  Fuzzy  number,  is  commonly  used  in  applications  of  Fuzzy 
mathematics.  In  the  present  application,  the  conclusions  to  be  drawn  will,  in  most  cases,  be 
relatively  insensitive  to  the  gauge  chosen  to  interpolate  between  the  inner  and  outer  intervals  of 
the  Fuzzy  results. 

Thus,  referring  to  Figure  A-2,  to  find  an  interval  that  corresponds  to  the  80%  confidence  used  in  the 
point  estimate  evaluation,  we  will  take  an  interval  encompassing  two-fifths  of  the  space  between  the 
"Probably"  and  "Surely"  points. 
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APPENDIX  B: 


METHODOLOGY/CODE  FOR  EVALUATION  OF 
SINGLE  ROUND  KILL  PROBABILITY  (SRKP) 


Intentionally  left  blank. 
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In  the  Fuzzy  analysis  approach  to  evaluation  of  required  vulnerability  information  set  (RVIS)  quality, 
it  is  efficient  and  desirable  to  use  the  same  analytical  techniques  and  codes  that  are  already  being  used 
in  the  development  of  the  item.  For  an  area  weapon  such  as  a  grenade,  an  analytical  technique  such  as 
FULL  SPRAY  is  indicated  in  which  independent  blast  and  fragment  vulnerabilities  are  combined  to 
produce  a  combined  blast-fragment  kill  assessment.  For  the  Ballistic  Bicycle  example,  a  Formula 
Translation  (FORTRAN)  Code  was  written  that  implements  the  basic  concepts  of  FULL  SPRAY.  That 
code  is  described  in  this  appendix. 

(Actually,  two  codes  were  written,  as  discussed  in  Appendix  C.) 

First,  the  single  round  kill  probability  (SRKP)  depends  upon  the  probability  of  an  incoming  round 
landing  at  a  given  radius  from  the  bicycle  times  the  probability  that  such  a  round  will  result  in  a  kill  of 
the  target. 


SRKP=  5^  PjjiR)xPf.{R)  (B-1) 

RADII 

In  equation  B-1,  Pfj(R)  is  an  input  (the  grenade  delivery  accuracy).  the  kill  given  a  round  at 

/? ,  depends  upon  the  probabilities  of  blast  and  fragment  kill.  Treating  them  as  independent,  we  have: 

PicW-  >  -<1  (B-2) 

or,  if  dealing  only  with  crisp  numbers  (see  Appendix  C), 

P^(R)=Pj5.  (R)+Pji.  {R)-Pfc  {R)y-Pic  iR)  .  (B-3) 

*  ^Frag^  ^BlasT  '  ^Frag  ^  ’ 


In  equations  B-2  and  B-3,  ^  ®  fundamental  vulnerability  input  (the  traditional  "blast  curve"). 

The  radii  of  interest  are  shown  in  Figure  B-1. 


* 

Computer  Program  for  General  Full  Spray  Materiel  MAE  Computations,  Volume  I-User  Manual,  JMEM  Report,  61JTCG/ME- 
79-1-1. 
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In  this  and  in  most  current  analyses,  only  single-cell  kills  are  considered.  Therefore,  it  depends  upon 
the  probability  of  a  fragment  hitting  a  grid  cell  (refer  to  Figure  B-2)  times  the  probability  that  such  a 
fragment  would  kill  the  bicycle.  Assuming  independence  among  cells,  we  have 


P  V 

‘^Frag 


(B-4) 
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Figure  B-2.  The  Ballistic  Bicycle:  fragment  grid. 

Assuming  an  isotropic  distribution  of  fragments  from  the  grenade,  the  hit  probability  of  a  cell  (Phq 
is  given  by 

(B-5) 

Here,  Fragsisr  is  die  density  of  fragments  per  steradian  emitted  by  the  grenade,  I  is  the  fraction  of 
the  cell  occupied  by  the  major  bicycle  components  (corresponding  to  the  "Importance"  parameter  used  in 
the  Point  Estimate  Technique),  and  Cellsize  and  R  are  expressed  in  the  same  (linear)  unit. 

Returning  to  equation  B-4,  Pk(^^ii  ^  ^sumed  to  depend  upon  at  most  one  critical  component  per  cell. 
In  addition,  kill  probability  is  accorded  to  synergistic  effects  in  several  cells.  (Synergistic  cells  were  also 


Cellsize^ 

R 

/ 
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used  in  the  Point  Estimate  Technique.)  Assuming  independence  between  synergistic  and  component 
failures,  we  have 


^CeU  ^Component  ^Synergism 


(B-6) 


As  was  the  case  for  (P),  P*-  and  P*-  are  fundamental  vulnerability  inputs. 

^  Blast  ^Component  ^Synergism 

The  program  that  implements  the  previous  equations  consists  of  a  main  routine  and  four  subroutines. 
Inputs,  all  read  from  files,  include: 

•  Pf^R)  Array 

•  P^  .  Array  (Fuzzy) 

‘^Synergism 

•  Cell  data,  which  consists  of 

-  The  (major)  component  in  each  cell 

-  The  synergism,  if  any 

-  The  occupied  fraction  of  the  cell 

-  Total  number  of  fragments  in  the  round 

-  Cell  size  (in  meters) 

Because  of  the  structure  of  the  input  data  and  the  association  of  component  and  synergistic  data  with 
each  cell,  it  is  easy  to  change  data  values  pertaining  to  components,  s)mergisms,  and  blast  and  rerun  the 
SRKP  calculation.  Advantage  was  taken  of  this  feature  in  producing  the  results  presented  in  the  main 
body  of  this  report. 
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APPENDIX  C: 

OBSERVATIONS  ON  IMPLEMENTATION  OF  FUZZY  MATH  IN 
CONVENTIONAL  COMPUTER  CODES 
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Intentionally  left  blank. 
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The  definition  of  the  sum  of  two  Fuzzy  numbers,  and  P2,  is  given  by: 


PPi  +  p/0  =  [min(pp^(ac),  |ip2(y))]  (C-1) 

In  Equation  C-1,  ^p^t)  is  the  value  of  the  "membership  function"  at  value  t,  and  max^^^_j  is  the 

maximum  for  all  values  of  x  and  y  such  that  x  +  y  =  t.  For  simple  (trapezoidal)  Fuzzy  numbers,  the 
interpretation  of  equation  refadd  is  quite  straightforward.  Referring  to  Figure  C-1,  one  sees  that  the 
smallest  value  of  t  for  which  the  minimum  of  Pp^(x)  and  «p^(y)  is  not  zero  must  be  at  c;  +  02-  Hence, 

a  1+2  -  ^1  +  ®2-  Similar  reasoning  gives  the  values  for  bj^2>  ^i+2’  ^7 +2* 

•  c^^2  =  +  ^2- 

•  ^7+2  =  ^7  +  h- 

•  Ci^2  =  ^7  ^2- 

•  ^1+2  =  dj  +  d2. 
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Very  similarly,  multiplication  for  trapezoidal  Fuzzy  numbers  can  be  simplified  to: 

•  "ix2  =  «2- 

•  *7x2  =  *7  X  *2- 

•  ^7x2  =  X  ^2- 

•  ^7x2  =  ^7  ^  ^2- 

Note  the  extremely  handy  implications  of  this  result.  As  long  as  the  only  operations  being  performed 
are  addition  and  multiplication  of  trapezoidal  Fuzzy  numbers,  there  is  no  mixing  of  the  respective  points 
of  the  Fuzzy  numbers:  lowest  values  are  combined  with  lowest  values,  lower  middles  with  lower  middles, 
etc.  This  implies  that  a  computer  code  which  involves  only  addition  and  multiplication  can  be  used  for 
Fuzzy  calculations  even  if  the  code  was  written  to  operate  only  on  crisp  (non-Fuzzy)  numbers:  One  need 
only  execute  the  code  fom  times  with  four  sets  of  input  data  corresponding  to  the  lowest  value  of  each 
datum,  the  lower  middle  values,  etc. 

Unfortunately,  this  extremely  fortuitous  simplification  does  not  pertain  if  other  operations  (such  as 
Fuzzy  subtraction)  are  included.  As  seen  in  Figure  C-2,  the  corresponding  rules  for  subtraction  are: 

•  ^7-2  ~  ~  ^2‘ 

•  *7-2  =  *7  -  «'2- 

•  Ci_2  =  Cj  -  b2. 

•  di_2  =  dj  -  ^2* 

And  similarly,  division  of  trapezoidal  Fuzzy  numbers  can  be  simplified  to; 

•  0^/2  =  aild2- 

•  bj,2  =  bj/C2. 

•  Cjj2  =  Cjlb2. 

•  dji2  =  djla2. 
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Figure  C-2.  Fuzzy  subtraction. 

Note  that  the  previous  discussion  refers  only  to  those  operations  involving  Fuzzy  numbers;  scalar 
arithmetic  involved  in  the  code  is  unchanged,  of  course. 

The  savings  that  could  be  realized  by  using  current  (crisp  number)  computer  codes  for  Fuzzy  analyses 
are  so  significant  that  one  is  led  to  investigate  how  values  which  are  potentially  Fuzzy  are  used  in 
common  codes.  To  this  end,  the  crisp-number  version  of  the  simple  computer  code  (about  500  lines) 
written  for  the  analysis  of  the  Ballistic  Bicycle,  was  analyzed  for  appropriateness.  It  was  found  that  the 
Fuzzy  data  (blast,  component,  and  synergistic  Pf^s)  were  involved  only  in  addition,  multiplication,  and  in 
"survivor  ruling"  of  the  form: 
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Q  =  1  -  [(1  -  A)  X  (1  -  B)J 


(C-2) 


Analysis  of  Equation  C-2  reveals  that,  in  spite  of  the  presence  of  subtraction,  the  respective  points  in 
the  Fuzzy  numbers  remained  unmixed:  both  A  and  B  are  reversed  before  multiplication,  and  the  outer 
negation  restores  the  order  that  was  switched  by  the  negations  inside  the  parentheses. 

[Note,  however,  that  the  "Survivor  Rule"  does  not  survive  the  expansion  shown  in  Equation  B-3. 
Were  Fuzzy  numbers  substituted  directly  into  the  expansion,  the  result  would  be  to  mix  the  values. 
(Subtraction  of  the  product  would  mix  a  product  of  highest  values  with  the  lowest  values  from 
the  first  two  terms.)  In  effect,  the  use  of  crisp  mathematics  to  expand  Equation  B-2  violates  the 
rules  of  Fuzzy  mathematics  an  odd  number  of  times.] 

To  verify  this  finding,  a  second  version  of  the  simple  Ballistic  Bicycle  analysis  code  was  written.  In 
this  version,  all  Fuzzy  numbers  are  treated  as  four-vectors,*  and  all  arithmetic  is  done  by  subroutines  that 
implement  die  rules  listed  previously.  As  expected,  the  Ballistic  Bicycle  results  via  the  second  version 
were  identical  to  those  obtained  by  four  runs  of  the  first  (crisp)  version. 

Clearly,  this  exercise  provides  no  proof  that  any  other  common  vulnerability  code  can  be  used  for 
Fuzzy  analyses  without  modification.  However,  given  that  the  operations  performed  for  the  Ballistic 
Bicycle  are  indeed  representative  of  diose  normally  performed,  there  is  reason  to  further  investigate  the 
common  codes. 


^  *  > 

Actually,  five-vectors.  In  addition  to  the  Fuzzy  arithmetic  involving  the  inner  and  outer  intervals,  "crisp"  calculations  were 

conducted  using  the  mean  (center)  values  of  all  input  variables.  The  results  of  these  calculations  were  used  as  the  basis  for 

the  evaluation  of  percentages. 
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APPENDIX  D: 


AN  EXERCISE  IN  BINARY  TESTING 


Intentionally  left  blank. 
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The  Ballistic  Bicycle  example  required  the  generation  of  a  hypothetical  test  plan  which  involved  only 
"full-up"  shots  (i.e.,  shots  in  which  the  weapon  [a  grenade]  was  employed  against  a  full  target  [bicycle]). 
Specifically,  the  example  required  an  estimation  of  the  confidence/accuracy  of  conclusions  that  could  be 
drawn  from  a  test  series. 

Although  just  an  exercise  to  generate  a  hypothetical  example,  the  exercise  proved  to  be  illustrative  of 
considerations  that  must  be  entertained  in  the  planning  of  an  actual  suite  of  tests.  We  have  therefore 
included  an  account  of  the  exercise  in  this  appendix. 

In  the  example,  it  is  stated  that  the  data  to  be  used  by  the  "decision  makers"  who  are  being  supported 
by  the  Ballistic  Bicycle  study  are  single  shot  kill  probabilities  (SRKPs).  We  take  SRKPs  to  be  given  by: 

SRKP  =  Jp(r)  X  P^(r)  dr,  (D-1) 

where  P(r)  is  the  probability  of  a  grenade  hitting  the  ground  at  a  particular  point  indicated  by  r,  Pjj(r)  is 
the  probability  of  a  kill  given  a  hit  at  r,  and  the  integral  is  taken  over  all  r. 

It  was  furthermore  given  that  the  accuracy  of  the  grenade  was  characterized  by  a  circular  error 
probable  (CEP)  of  6  m. 

We  define  the  fractional  error  in  SRKP  as: 


ASRKP  =  J  P(r)  X 


APk(r) 

Pk(r) 


dr. 


(D-2) 


where  AP^(r)  is  the  uncertainty  in  the  probability  of  a  kill  given  a  hit  at  r.  It  is  this  uncertainty  that  we 
intend  to  decrease  through  Test  Plan  3. 

The  next  step  in  building  up  this  example  was  to  generate  uncertainty  estimates.  For  this  purpose, 
we  used  the  binomial  distribution  as  a  data  generator.  In  particular,  if  p  is  the  probability  of  a  kill, 
then — ^in  N  shots — the  expected  number  of  kills,  x,  is  given  by 

X  =  N  X  p. 


131 


Assuming  that  the  results  come  from  a  binomial  distribution,  the  standard  deviation  in  x  is  given  by: 


Ax  =  \/Np(l-p)  , 

and  the  fractional  uncertainty  in  the  kill  probability  (p)  is  given  by: 

APj.(r)  =  V  p(l-p)/N  .  (D-3) 

It  is  also  necessary  to  adopt  a  test  strategy,  preferably  one  that  will  minimize  the  cost  of  the  test  series. 
One  such  strategy  is  to  begin  with  distant  shots  that  are  not  expected  to  damage  a  target.  One 
systematically  moves  in  until  lolls  begin  to  occur.  At  this  point,  sufficient  shots  are  taken  at  each  point 
to  bring  the  imcertainty,  given  by  equation  D-3,  to  the  desired  size  before  proceeding  inward  another  step. 
Once  reaching  the  range  of  sure  kills,  the  series  stops. 

The  uncertainty  in  the  SRKP,  given  by  equation  D-2,  is  then  found  by  multiplying  the  probability  of 
hitting  at  each  distance  from  the  bicycle — as  calculated  from  the  CEP — ^times  the  associated 
imcertainty — ^given  by  equation  D-3. 

TTie  results  of  this  exercise  are  shown  in  Table  D-1 .  For  each  distance,  the  circular  hit  probability  was 
evaluated  and  the  kill  probability  estimated.  Then,  we  determined  the  number  of  shots  required  to  bring 
down  the  contribution  at  that  miss  distance  to  that  overall  uncertainty  (last  column  in  Table  D-1).  Finally, 
the  total  uncertainty  was  found  by  integrating  (summing)  over  the  miss  distances,  as  indicated  by 
equation  D-2. 
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Table  D-1.  Bin  Estimation  for  Test  Plan  3 


Distance 
From  Target 
(m) 

P(r) 

Shots 

Kills 

P 

(est.) 

Frac.  s.d. 

dPk(r)*P(r) 

6 

0.12 

1 

1 

0.000 

7 

0.10 

5 

4 

0.800 

0.179 

0.018 

8 

0.09 

10 

5 

0.500 

0.158 

0.014 

9 

0.07 

6 

2 

0.333 

0.192 

0.013 

10 

0.06 

5 

1 

0.200 

0.179 

0.011 

11 

0.04 

1 

0 

0.000 

0.000 

0.000 

12 

0.03 

1 

0 

0.000 

0.000 

0.000 

13 

0.02 

1 

0 

0.000 

0.000 

0.000 

TOTALS 

30 

13 

0.056 
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LIST  OF  ABBREVIATIONS 


ABDR 

Aircraft  battle  damage  and  repair 

ACAT 

Acquisition  Category 

AFAS 

Advanced  Field  Artillery  System 

ALF 

Army  live-fire 

AMSAA 

U.S.  Army  Materiel  Systems  Analysis  Activity 

ARL 

U.S.  Army  Research  Laboratory 

ASB 

Air  Systems  Branch 

BAD 

Behind-armor  debris 

BDAR 

Battle  damage  assessment  and  repair 

BDR 

Battle  damage  repair 

BVLD 

Ballistic  Vulnerability/Lethality  Division 

CASTFOREM  - 

Combined  Arms  and  Support  Task  Force  Evaluation  Model 

CDR 

Critical  Design  Review 

CEP 

Circular  Error  Probable 

COEA 

Cost  and  operational  effectiveness  analysis 

CVG 

Committee  on  Vulnerability  Guidelines 

DOD 

Department  of  Defrase 

DODD 

Department  of  Defense  directive 

DODI 

Department  of  Defense  instruction 

DSVM 

Degraded  states  vulnerability  methodology 

DT&E/ASP  - 

Director,  Test  and  Evaluation,  Air  and  Space  Program 

DT&E/LMS  - 

Director,  Test  and  Evaluation,  Land  and  Maritime  Systems 

DT  and  OT  - 

Development  tests  and  operational  tests 

DUSA(OR)  - 

Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

DVC 

Damaged  vehicle  cost 

EMD 

Engineering  Manufacturing  Development 

EMV 

Expected  monetary  value 

FFL 

Fraction  of  the  fleet  lost 

FORTRAN  - 

Formula  Translation  Code 

HEI 

High-explosive  incendiary 

JLF 

Joint  live  fire 

135 


JTCG 

- 

Joint  Technical  Coordinating  Group 

JTCG/AS 

- 

Joint  Technical  Coordinating  Group  on  Aircraft  Survivability 

KE 

- 

Kinetic  energy 

LF 

Live-fire 

LFT 

- 

Live-fire  testing 

LFT&E 

- 

Live  Fire  Test  and  Evaluation 

LOF 

- 

Loss  of  function 

LRIP 

- 

Low  Rate  Initial  Production 

MNS 

- 

Mission  need  statement 

MS  n 

- 

Milestone  n 

NRC 

- 

National  Research  Coimcil 

ORD 

- 

Operational  requirements  document 

OSD 

- 

Office  of  the  Secretary  of  Defense 

PDR 

- 

Preliminary  Design  Review 

PMS 

- 

Premilestone 

RAM 

- 

Reliability,  availability,  and  maintainability 

RAMD 

- 

Reliability,  Availability,  and  Maintainability  Division 

RVIS 

- 

Required  vulnerabDity  information  set 

SARDA 

- 

Assistant  Secretary  of  the  Army  (Research,  Development,  and  Acquisition) 

SC 

Shaped-charge 

SDAL 

- 

Standard  damage  assessment  list 

SDIO 

“ 

Strategic  Defense  Initiative  Office 

SLAD 

- 

Survivability/Lethality  Analysis  Directorate 

SOW 

- 

Scope-of-work 

SQuASH 

- 

Stochastic  Quantitative  Analysis  of  System  Hierarchies 

SRKP 

- 

Single  round  kill  probability 

SSKP 

- 

Single  shot  kill  probability 

STAR 

- 

System  Threat  Assessment  Report 

T&E 

- 

Test  and  evaluation 

TEMP 

- 

Test  and  Evaluation  Master  Plan 

TRAC-WSMR 

- 

TRADOC  Analysis  Center  at  White  Sands  Missile  Range 

TRADOC 

- 

U.S.  Army  Training  and  Doctrine  Command 

VAMP 

- 

Vulnerability  Analysis  Methodology  Program 
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VAST 

V/L 

VRM 

WPAFB 


-  •  Vulnerability  analysis  for  surface  targets 
Vulnerability/lethality 
Vulnerability  reduction  measures 
Wright-Patterson  Air  Force  Base 
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NO.  OF 
COPIES 


ORGANIZATION 


2  DEFENSE  TECHNICAL  INFO  CTR 
ATTN  DTIC  DDA 

8725  JOHN  J  KINGMAN  RD 
STE0944 

FT  BELVOIR  VA  22060-6218 

1  HQDA 

DAMOFDQ 

ATTN  DENNIS  SCHMIDT 
400  ARMY  PENTAGON 
WASHINGTON  DC  20310-0460 

1  US  MILITARY  ACADEMY 

MATH  SCI  CTR  OF  EXCELLENCE 
DEPT  OF  MATHEMATICAL  SCI 
ATTN  MDN  A  MAJ  DON  ENGEN 
THAYER  HALL 
WEST  POINT  NY  10996-1786 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
ATTNAMSRLCSALTP 
2800  POWDER  MILL  RD 
ADELPHI MD  20783-1 145 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
ATTN  AMSRL  CS  AL  TA 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 145 

3  DIRECTOR 

US  ARMY  RESEARCH  LAB 
ATTN  AMSRL  Cl  IX 
2800  POWDER  MUX  RD 
ADELPHI  MD  20783-1 145 


ABERDEEN  PROVING  GROUND 

2  DIRUSARL 

ATTN  AMSRL  Cl  LP  (305) 
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NO.  OF 
COPIES 

1 


1 


1 


1 


1 


1 


1 


1 


ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


OSD  OUSD  AT 

STRT  TAC  SYS 

ATTNDRSCHNEITER 

3090  DEFNS  PENTAGON  RM  3E130 

WASHINGTON  DC  20301-3090 

ASST  SECY  ARMY  RESEARCH 
DEVELOPMENT  ACQUISITION 
ATTN  SARD  ZD  RM  2E673 
103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 


1  ARMY  RESEARCH  LABORATORY 

ATTNAMSRLSL 
PROGRAMS  AND  PLANS  MGR 
WSMR  NM  88002-5513 

1  ARMY  RESEARCH  LABORATORY 

ATTN  AMSRL  SL  E 
MR  MARES 
WSMR  NM  88002-5513 

ABERDEEN  PROVING  GROUND 


ASST  SECY  ARMY  RESEARCH 
DEVELOPMENT  ACQUISITION 
ATTN  SARD  ZP  RM  2E661 
103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 


1  CDRUSATECOM 
ATTN:  AMSTE-TA 


2  DIR  USAMSAA 

ATTN:  AMXSY-ST 
AMXSY-D 


ASST  SECY  ARMY  RESEARCH 

DIRUSARL 
ATTN:  AMSRL-SL, 

J  WADE  (433) 

M  STARKS  (433) 

AMSRL-SL-C,  J  BEILFUSS  (E3331) 
AMSRL-SL-B,  P  DEITZ  (328) 

DEVELOPMENT  ACQUISITION 
ATTN  SARD  ZT  RM  3E374 
103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 


DEVELOPMENT  ACQUISITION  4 

ATTN  SARD  ZS  RM  3E448 
103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 

ASST  SECY  ARMY  RESEARCH 


UNDER  SEC  OF  THE  ARMY 

ATTNDUSAOR 

RM2E660 

102  ARMY  PENTAGON 
WASHINGTON  DC  20310-0102 


ASST  DEP  CHIEF  OF  STAFF 
OPERATIONS  AND  PLANS 
ATTN  DAMO  FDZ  RM  3A522 
460  ARMY  PENTAGON 
WASHINGTON  DC  20310-0460 

DEPUTY  CHIEF  OF  STAFF 
OPmiATIONS  AND  PLANS 
ATTN  DAMO  SW  RM  3C630 
400  ARMY  PENTAGON 
WASHINGTON  DC  20310-0400 
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No  of 

Copies  Organization 

1  DEP  ASST  SEC  DEF 

CHEM  BIO  MATTERS 
ATTN  DR  PROCIV 
2451  CRYSTAL  PARK  DR 
ALEXANDRIA  VA  22245 

1  DEFENSE  MAPPING  AGENCY  HQ 
TECH  AND  INFO  DIRECTORATE 
8613  LEE  HIGHWAY 
ATTN  THOMAS  HENNING 
FAIRFAX  VA  22031  -2137 

1  BALLISTIC  MISSILE  DEFENSE 
ORGANIZATION 
ATTN  DEPUTY  TECHNOLOGY 
RM  1  El  48  COL  GARY  PAYTON 
THE  PENTAGON 
WASHINGTON  DC  20310-7100 

1  BALLISTIC  MISSILE  DEFENSE 
ORGANIZATION  ATTN  DTC 
RM  1  El  68  DR  WALTER  DYER 
THE  PENTAGON 
WASHINGTON  DC  20310-7100 

1  BALLISTIC  MISSILE  DEFENSE 
ORGANIZATION 
ATTN  DTE  LTCDALETIETZ 
THE  PENTAGON 
WASHINGTON  DC  20310-7100 

1  DEFENSE  NUCLEAR  AGENCY 
ATTN  DEPUTY  DIRECTOR 
DR  GEORGE  ULLRICH 
6801  TELEGRAPH  RD 
ALEXANDRIA  VA  2231 0-3398 

1  DEFENSE  NUCLEAR  AGENCY 
ATTN  OFTA 
DR  MICHAEL  SHORE 
6801  TELEGRAPH  RD 
ALEXANDRIA  VA  2231 0-3398 

1  DEPUTY  CHIEF  OF  STAFF 
INTELLIGENCE 
ATTN  DAMI  ST 
DR  BERT  SMITH 
1001  ARMY  PENTAGON 
WASHINGTON  DC  20310-1001 

1  DIR  OPERATIONAL  TEST  &  EVAL 
1700  DEFENSE  PENTAGON 
RM  3E318 

ATTN  PHILIP  E  COYLE 
WASHINGTON  DC  20301-1700 


No  of 

Copies  Organization 

1  DIR  OPERATIONAL  TEST  &  EVAL 
1700  DEFENSE  PENTAGON 
RM  3E318 

ATTN  ERNEST  SEGLIE 
WASHINGTON  DC  20301-1700 

1  DIR  OPERATIONAL  TEST  &  EVAL 
1700  DEFENSE  PENTAGON 
RM  1C730A 

ATTN  JAMES  F  O’BRYON 
WASHINGTON  DC  20301-1700 

1  DIR  OPERATIONAL  TEST  &  EVAL 
1700  DEFENSE  PENTAGON 
ATTN  HANS  MAIR 
WASHINGTON  DC  20301-1700 

1  DIR  OPERATIONAL  TEST  &  EVAL 
1 700  DEFENSE  PENTAGON 
RM  1C730A 

ATTN  THOMAS  R  JULIAN 
WASHINGTON  DC  20301-1700 

1  DIR  OPERATIONAL  TEST  &  EVAL 
1700  DEFENSE  PENTAGON 
RM  1C730A 

ATTN  COL  STEPHEN  C  DALY 
WASHINGTON  DC  20301-1700 

1  ODDRNE  AT 

DIR  WEAPONS  TECH 
ATTN  WES  KITCHENS 
3080  DEFENSE  PENTAGON 
WASHINGTON  DC  20301  -3080 

1  UNDER  SECY  OF  THE  ARMY 
ATTN  SAUS  OR 
DANIEL  WILLARD 

102  ARMY  PENTAGON 
WASHINGTON  DC  20310-0102 

1  ASST  SECY  ARMY  RESEARCH 
DEVELOPMENT  ACQUISITION 
ATTN  SARD  ZD  RM  2E673 
DR  HERBERT  FALUN 

103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 

1  ASST  SECY  ARMY  RESEARCH 
DEVELOPMENT  ACQUISITION 
ATTN  SARD  TR  RM  3E480 
TOM  KILLION 
103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 
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No  of 

Copies  Organization 

1  ASST  SECY  ARMY  RESEARCH 
DEVELOPMENT  ACQUISITION 
ATTN  SARD  DOV  RM  2E673 
COL  CORLISS  S  BERRY  III 
103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 

1  PROGRAM  EXECUTIVE  OFFICER 
ARMORED  SYS  MODERNIZATION 
ATTN  SFAE  ASM 
WARREN  Ml  48397-5000 

1  PROGRAM  EXECUTIVE  OFFICER 
TACTICAL  WHEELED  VEHICLES 
ATTN  SFAE  TWV 
COL  WILLIAM  J  STODDART 
WARREN  Ml  48397-5000 

1  PROG  EXECUTIVE  OFCR 
AVIATION 
ATTN  SFAE  AV 
MG  DEWITT  T  IRBY  JR 
4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1798 

1  PROG  EXECUTIVE  OFCR 
MISSILE  DEFENSE 
ATTN  SFAE  MD 
BG  DANIEL  L  MONTGOMERY 
PO  BOX  16686 

ARLINGTON  VA  2221 5 - 1 686 

1  PROG  EXECUTIVE  OFCR 
FIELD  ARTILLERY  SYS 
ATTN  SFAE  FAS 
BG  JOHNFMICHITSCH 
PICATINNY  ARSENAL  NJ  07806-5000 

1  PROG  EXECUTIVE  OFCR 
TACTICAL  MISSILES 
ATTN  SFAE  MSL 
MR  GEORGE  G  WILLIAMS 
REDSTONE  ARSENAL  AL  35898-8000 

1  PROJECT  MANAGER 
PEO  MISSILE  DEFENSE 
ATTN  SFAE  MD  PA 
COL  STEPHEN  KUFFNER 
PO  BOX  1500 

HUNTSVILLE  AL  35807-3801 


No  of 

Copies  Organization 

1  PROJECT  MANAGER 
PEO  MISSILE  DEFENSE 
ATTN  SFAE  MDTHA 
COL  WALTER  F  KILGORE 
PO  BOX  1500 

HUNTSVILLE  AL  35807-3801 

1  PRODUCT  MANAGER 

PEO  INTELL  ELEC  WARFARE 
ATTN  SFAE  lEW  RD 
LTC  JAMES  WELLS 
REDSTONE  ARSENAL  AL  35898 

1  PROJECT  MANAGER 
PEO  MISSILE  DEFENSE 
ATTN  SFAE  MD  SM 
COL  THOMAS  HALLER 
PO  BOX  1500 

HUNTSVILLE  AL  35807-3801 

1  PRODUCT  MANAGER 
PEO  TACTICAL  MISSILES 
ATTN  SFAE  MSL  FAD  GTAM 
LTC  JED  A  SHEEHAN 
REDSTONE  ARSENAL  AL  35898-5630 

1  PROGRAM  MANAGER 
ATTN  SFAE  AV  RAH 
BLDG  105 

4300  GOODFELLOW  BLVD 
BG  JAMES  SNIDER 
ST  LOUIS  MO  631 20-1798 

1  PROJECT  MANAGER 

ATTN  SFAEAVAAH 
BLDG  105 

4300  GOODFELLOW  BLVD 
COL  STEPHEN  KEE 
ST  LOUIS  MO  631 20- 1 798 

1  PRODUCT  MANAGER 
ATTN  SFAE  AVSOA 
LTC  MICHAEL  ROGERS 
BLDG105F 

4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1798 

1  PROJECT  MANAGER 
ATTN  SFAE  AV  ASH 
COL  EDWIN  GOOSEN 
BLDG  102 

4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1798 
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1  PRODUCT  MANAGER 
ATTN  AMCPM  CH 
JAMES  WINKELER 
4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1798 

1  CMDR  US  ARMY  AVN  TROOP  CMND 
TECHNLGY  APPLCTN  PROG  OFC 
ATTNAMSATDTS 
LTC  WAYNE  KILLIAN 
4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1798 

1  PRODUCT  MANAGER 
ATTNSFAEAVLB 
LTC  HOWARD  BRAMBLETT 
BLDG  105 

4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1789 

1  PRODUCT  MANAGER 
ATTN  SFAE 
LTC  RICHARD  RYLES 
BLDG  105 

4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1798 

1  PROJ  MANAGER  CRUSADER 
PEO  FIELD  ARTILLERY  SYS 
ATTN  SFAE  FAS  CR 
COL  WILLIAM  B  SHEAVES  III 
PICATINNY  ARSENAL  NJ  07806-5000 

1  PROJ  MANAGER  HTV 

PEO  TACTICAL  WHEELED  VEH 
ATTN  SFAE  TWV  HTV 
COL  JAMES  WANK 
WARREN  Ml  48397-5000 

1  PROJECT  MANAGER  BFVS 
PEO  ASM 

ATTN  SFAE  ASM  BV 
LTC  TED  JOHNSON 
WARREN  Ml  48397-5000 

1  PROD  MANAGER  BFVS  C2V 
PEO  ASM 

ATTN  SFAE  ASM  BV 
LTC  PAUL  WILSON 
WARREN  Ml  48397-5000 

1  PROD  MANAGER  ABRAMS 
PEO  ASM 

ATTN  SFAE  ASM  AB 
LTC  GEORGE  PATTEN 
WARREN  Ml  48397-5000 


No  of 

Copies  Organization 

1  PROD  MANAGER  HAB 
PEO  ASM 

ATTN  SFAE  ASM  CV 
LTC  SAMUEL  M  CANNON 
WARREN  Ml  48397-5000 

1  PROD  MANAGER  Ml  BREACHER 
PEO  ASM 

ATTN  SFAE  ASM  CV 
LTC  DONALD  P  KOTCHMAN 
WARREN  Ml  48397-5000 

1  PROD  MGR  PALADIN  FAASV 
PEO  FIELD  ARTILLERY  SYS 
ATTN  SFAE  FAS  PAL 
LTC  CHARLES  CARTWRIGHT 
PICATINNY  ARSENAL  NJ  07806-5000 

1  PROJ  MGR  FMTV 

PEO  TACTICAL  WHEELED  VEHICLES 
ATTN  SFAE  TWV  FMTV 
COL  MICHAEL  W  BOUDREAU 
WARREN  Ml  48397-5000 

1  PROD  MGR  IRV 
PEO  ASM 

ATTN  SFAE  ASM  CV 
LTC  ROBERTS  LEES  JR 
WARREN  Ml  48397-5000 

1  PROJ  MGR  MLRS 

PEO  TACTICAL  MISSILES 
ATTN  SFAE  MSL  ML 
COL  STEVEN  WFLOHR 
REDSTONE  ARSENAL  AL  35898-5700 

1  PROJ  MGR  LTV 

PEO  TACTICAL  WHEELED  VEH 
ATTN  SFAE  TWV  LTV 
LTC  BRAD  R  NAEGLE 
WARREN  Ml  48397-5000 

1  PROJ  MGR  HDSB 

PEO  TACTICAL  WHEELED  VEH 
ATTN  SFAE  TWV  HTV 
COL  JAMES  A  WANK 
WARREN  Ml  48397-5000 

1  PROJ  MGR  HMMWV 

PEO  TACTICAL  WHEELED  VEH 
ATTN  SFAE  TWV  HTV 
JOHN  D  WEAVER 
WARREN  Ml  48397-5000 
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1  PROJECT  MGR  LW  155 
PEO  FIELD  ARTILLERY  SYS 
ATTN  SFAE  FAS  J 
COL  STEPHEN  C  WARD 
PICATINNY  ARSENAL  NJ  07806-5000 

1  PROJ  MGR  MCD 
PEO  ASM 

ATTN  SFAE  ASM  MCD 

COL  RICHARD  DNIDEL 

PICATINNY  ARSENAL  NJ  07806-5000 

1  PROJ  MGR  ARMY  TACMS  BA 
PEO  TACTICAL  MISSILES 
ATTN  SFAE  MSLAB 
LTC  KEITH  LENHARD 
REDSTONE  ARSENAL  AL  35898-5650 

1  PROJ  MGR  ARMY  TACMS  BA 
PEO  TACTICAL  MISSILES 
ATTN  SFAE  MSLAB 
LTC  BARRY  WARD 

REDSTONE  ARSENAL  AL  35898-5650 

1  PROJ  MGR  ARMY  TACMS  BA 
PEO  TACTICAL  MISSILES 
ATTN  SFAE  MSLAB 
'  LTC  ROBERT  ARNONE 

REDSTONE  ARSENAL  AL  35898-5650 

1  PROJ  MGR  ARMY  TACMS  BA 
PEO  TAOTICAL  MISSILES 
ATTN  SFAE  MSLAB 
FRANK  WALLACE 

REDSTONE  ARSENAL  AL  35898-5650 

1  PROD  MGR  LB  HELLFIRE 
PEO  TACTICAL  MISSILE 
ATTN  SFAE  MSL  HD 
LTC  DONALD  WILBOURN 
REDSTONE  ARSENAL  AL  35898-561 0 

1  PROJ  MGR  JAVELIN 
PEO  TACTICAL  MISSILES 
ATTN  SFAE  MSL  AM 
COL  MICHAEL  RODDY  III 
REDSTONE  ARSENAL  AL  35898-5720 

1  PROJ  MGR  MLRS 

PEO  TACTICAL  MISSILES 

ATTN  SFAE  MSL  ML 

COL  STEVEN  FLOHR 

REDSTONE  ARSENAL  AL  35898-5700 


No  of 

Copies  Organization 

1  PROJ  MGR  SADARM 

PEO  FIELD  ARTILLERY  SYS 
ATTN  SFAE  FAS  SD 
COL  JAMES  E  UNTERSEHER 
PICATINNY  ARSENAL  NJ  07806-5000 

1  PROJ  MGR  TANK  MAIN 

ARMAMENT  SYSTEMS  TMAS 
PEO  ASM 

ATTN  SFAE  ASM  TMA 
COL  RICHARD  BREGARD 
PICATINNY  ARSENAL  NJ  07806-5000 

1  PROJ  MGR  CLOSE  COMBAT 
ANTI  ARMOR  WEAPON  SYSTEM 
PEO  TACTICAL  MISSILES 
ATTN  SFAE  MSL  CC 
COL  ROBERT  ARMBRUSTER 
REDSTONE  ARSENAL  AL  35898-5710 

1  PROJ  MGR  NON  LINE  OF  SIGHT 
COMBINED  ARMS  NLOSCA 
PEO  TACTICAL  MISSILES 
ATTN  SFAE  MSL  NL 
COL  ROY  MILLAR 

REDSTONE  ARSENAL  AL  35898-5793 

1  PROD  MGR 
USAARDEC 
ATTN  AMCPM  SA 
LTC  JAMES  CAMBRON 
PICATINNY  ARSENAL  NJ  07806-5000 

1  PROJECT  MANAGER 

ARMORED  SYS  INTEGRATION 
PEO  ASM 

ATTN  SFAE  ASM  AS 

COLNEWLIN 

WARREN  Ml  48397-5000 

1  COMMANDER 

US  ARMY  MATERIEL  CMND 
DEP  CHF  OF  STAFF  FOR  RDE 
ATTN  AMCRD 
5001  EISENHOWER  AVE 
ALEXANDRIA  VA  22333-0001 

2  COMMANDER 

US  ARMY  MATERIEL  CMND 
ATTN  AMCRD  IT 
DRPAULEHLE 
MR  MICHAEL  DAILEY 
5001  EISENHOWER  AVE 
ALEXANDRIA  VA  22333-0001 
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1  CMDR  US  ARMY  MATERIEL  CMND 

PRINCIPAL  DEP  FOR  ACQSTN 
ATTNAMCDCGA 
MR  DALE  G  ADAMS 
5001  EISENHOWER  AVE 
ALEXANDRIA  VA  22333-0001 

1  CMDR  US  ARMY  MATERIEL  CMND 

PRINCIPAL  DEPUTY  FOR  TECH 
ATTN  AMCDCG  T 
MR  MICHAEL  FISETTE 
5001  EISENHOWER  AVE 
ALEXANDRIA  VA  22333-0001 

1  COMMANDER 
USARMYATCOM 
ATTNAMSATG 
4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1798 

1  COMMANDER 

US  ARMY  AVN  TROOP  CMND 
AVIATION  APPL  TECH  DIR 
ATTNAMSATRTV 
FORT  EUSTIS  VA  23604-5577 

1  COMMANDER 

USA  SIM  TRNG  INSTRU  CMND 
ATTNAMCP  DIS 
COL  JAMES  ETCHECHURY 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  TECHNICAL  DIRECTOR 

USAARDEC 
ATTNSMCARTD 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  ARMAMENTS  RDEC 
ATTN  SMCARASM 
FREDSCHUHMANN 
PICATINNY  ARSENAL  NJ  07806-5000 

1  TECHNICAL  DIRECTOR 
USA  AVN  TROOP  CMND 
ATTN  AMSAT  R  ZT 
ROBERT  KENNEDY 
4300  GOODFELLOW  BLVD 
ST  LOUIS  MO  63120-1798 

1  TECHNICAL  DIRECTOR 

USA  MISSILE  CMND  RDE  CTR 
ATTNAMSMI  RD 
DR  WILLIAM  MCCORKLE 
REDSTONE  ARSENAL  AL  35898-5240 


No  of 

Copies  Organization 

1  COMMANDER 

USA  TANK  AUTO  CMND  RDEC 
ATTNAMSTATR 
WAYNE  WHEELOCK 
WARREN  Ml  48397-5000 

1  COMMANDER 

USA  TANK  AUTO  CMND  RDEC 
ATTNAMSTATRD 
COLSCHREPPLE 
WARREN  Ml  48397-5000 

1  COMMANDER 

USA  TANK  AUTO  CMND  RDEC 
ATTNAMSTATRR 
DR  JACK  PARKS 
WARREN  Ml  48397-5000 

1  ARMY  RESEARCH  LABORATORY 
ATTN  AMSRL  D 
DR  JOHN  LYONS 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  ARMY  RESEARCH  LABORATORY 
SURV  LETH  ANALYSIS  DIR 
ATTN  AMSRL  SL 
LTC  MALONEY 
WSMR  NM  88002-5513 

1  ARMY  RESEARCH  LABORATORY 
SURV  LETH  ANALYSIS  DIR 
ATTN  AMSRL  SLCA 
DR  ROBERT  SUTHERLAND 
WSMR  NM  88002-5513 

5  ARMY  RESEARCH  LABORATORY 
SURV  LETH  ANALYSIS  DIR 
ATTN  AMSRL  SLEA 
R  FLORES  T  MCDONALD 
G  GUZIE  K  NIX  L  ESCUDERO 
WSMR  NM  88002-5513 

4  ARMY  RESEARCH  LABORATORY 
SURV  LETH  ANALYSIS  DIR 
ATTN  AMSRL  SLEV 
K  MORRISON  EZARRET 
J  CUELLAR  O  PAYAN 
WSMR  NM  88002-5513 

3  ARMY  RESEARCH  LABORATORY 
SURV  LETH  ANALYSIS  DIR 
ATTN  AMSRL  SLEP 
J  PALOMO  D  HUNT 
L ANDERSON 
WSMR  NM  88002-5513 


145 


No  of 

Copies  Organization 

2  ARMY  RESEARCH  LABORATORY 
SURV  LETH  ANALYSIS  DIR 
ATTNAMSRLSL  EG 
D  HEMMINGWAY 
R  PRICE 

WSMR  NM  88002-5513 

1  ARMY  RESEARCH  LABORATORY 
SURV  LETH  ANALYSIS  DIR 
ATTNAMSRLSL  El 
J  NOWAK 

FT  MONMOUTH  NJ  07703-5601 

1  ARMY  RESEARCH  LABORATORY 
SURV  LETH  ANALYSIS  DIR 
ATTNAMSRL  SLEU 
A  ESTORGA 
WSMR  NM  88002-5513 

1  ARMY  RESEARCH  LABORATORY 
ATTN  AMSRL  CB  PI 
T  WHITE 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  ARMY  RESEARCH  LABORATORY 
ATTN  AMSRL  PBP 
J  SCZEPANSKI 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  COMMANDER 

USATRNG  DOCTRINE  CMND 
ATTN  ATCD  M 
COL  VOLZ 

FT  MONROE  VA  23651  -5000 

1  COMMANDER 

USATRNG  DOCTRINE  CMND 
ATTN  ATCD  H 
COL  HARTMAN 
FT  MONROE  VA  23651  -5000 

1  DIRECTOR 

ARMY  FIELD  ARTILLERY  SCHOOL 
ATTNATSFFSC3A 
COL  RICHARD  SHERWOOD 
FT  SILL  OK  73503-5600 

1  DIRECTOR 

ARMY  FIELD  ARTILLERY  SCHOOL 
ATTNATSF  CN 
COL  BRISTOL  WILLIAMS 
FT  SILL  OK  73503-5600 


No  of 

Copies  Organization 

1  DIRECTOR 

ARMY  FIELD  ARTILLERY  SCH 
ATTN  ATSF  RMS 
COL  RICHARD  VALLARIO 
FT  SILL  OK  73503-5600 

1  DIRECTOR 

ARMY  INFANTRY  SCHOOL 
ATTN  ATZB  FS 
COL  JAMES  GRIBSCHAW 
FT  BENNING  GA  31905-5403 

1  DIRECTOR 

ARMY  INFANTRY  SCHOOL 
ATTN  ATZB  BV 
COL  THOMAS  STRAUSS 
FT  BENNING  GA  31905-5400 

1  DIRECTOR 

ARMY  AIR  DEFENSE 
ARTILLERY  SCHOOL 
ATTNATSATSMTM 
COLWILLIAMT  SMITH  III 
FT  BLISS  TX  7991 6-0002 

1  DIRECTOR 

ARMY ARMOR  SCHOOL 
ATTNATZKTS 
COL  JOHN  KALB 
FT  KNOX  KV  401 21 -5201 

1  DIRECTOR 

ARMY  ARMOR  CENTER 
ATTN  ATZK  SM 
COL  STEPHEN  MAIN 
FT  KNOX  K/ 401 21 -5000 

1  DIRECTOR 

ARMY  ARMOR  CTR 
ATTN  ATZK  XXI 
BLDG  1109 
MAJ  SHUFELT 
FT  KNOX  KY  401 21 -5201 

1  DIRECTOR 

ARMY  AVIATION  CTR 
ATTN  ATZQ  TSM  C 
COL  CHARLES  GANT  JR 
FT  RUCKER  AL  36362-501 0 

1  DIRECTOR 

ARMY  AVIATION  CTR 
ATTN  ATZQ  TSM  LB 
COL  ROBERT  MITCHELL 
FT  RUCKER  AL  36362-501 2 
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1  DIR  ARMY  ENGINEER  CTR 
AND  FT  LEONARD  WOOD 
ATTNATSETSM 
COL  COMADORA  FERGUSON 
FT  LEONARD  WOOD  MO  65473-5000 

1  DIRECTOR 

ARMY  TRANSPORTATION  SCH 
ATTNATZFTW 
COL  JAMES  WEBSTER  JR 
FT  EUSTIS  VA  23604-5457 

1  COMMANDER 
ARMYTRADOC 
ATTNATCDL 
LTC  RONALD  STEWART 
FT  MONROE  VA  23651  -5000 

1  COMMANDER 

US  ARMY  ARMOR  CTR 
ATTNATZKMW 
COL  DAVID  PORTER 
FT  KNOX  KY  40121  -5000 

1  DIRECTOR 

ARMY  TRADOC  ANALYSIS  CTR 
ATTNATRCWMB 

LUIS  DOMINGUEZ 
WSMR  NM  88002-5502 

1  DIRECTOR 

ARMY  TRADOC  ANALYSIS  CTR 

ATTNWS 

ALKIENTZ 

WSMR  NM  88002-5502 

1  COMMANDANT 

US  ARMY  INFANTRY  SCH 
ATTN  ATSH  CD 
COL  WILLIAM  PATTERSON 
FT  BENNING  GA  31905-5403 

1  COMMANDER 

USA  FIELD  ARTILLERY  CTR 
ATTNATSFCD 
COL  TONEY  STRICKLIN 
FT  SILL  OK  73503-5000 

1  COMMANDANT 
USA  ARMOR  CTR 
ATTN  ATSB  CD 
COL  EDWARD  BRYLA 
FT  KNOX  KY  401 21 -5000 


1  COMMANDANT 

US  ARMY  AVIATION  CTR 
ATTN  ATZQ  CD 
DAVEAHEARN 
FT  RUCKER  AL  36362-5000 

1  COMMANDANT 

US  ARMY  AIR  DEFENSE  CTR 
ATTNATSACD 
COL  MICHAEL  A  VANE 
FT  BLISS  TX  79916-0002 

1  DEPT  OF  NAVY 

DEP  DIR  NAVY  T&E  TECH  REQ 
ATTN  LEWIS  LUNBERG 
PENTAGON  RM  5C686 
WASHINGTON  DC  20350 

1  OFFICE  OF  NAVAL  RESEARCH 
TECH  AREA  MGR  SURF  AERO 
WEAPONRY  CODE  35 
ATTN  DAVID  SIEGEL 
800  N  QUINCY  ST 
ARLINGTON  VA  22217-5660 

1  NAVAL  AIR  SYSTEMS  CMD  HQ 

1421  JEFFERSON  DAVIS  HIGHWAY 
ATTN  DAVE  BUTLER 
ARLINGTON  VA  22243 

1  NAVAL  SURFACE  WARFARE  CTR 

CARDEROCK  DIVISION 
ATTN  FRED  FISCH 
BETHESDA  MD  20084-5000 

1  NAVAL  SURFACE  WARFARE  CTR 

DAHLGREN  DIVISION 
ATTN  G24 
TOM  WASMUND 
17320  DAHLGREN  RD 
DAHLGREN  VA  22448-51 00 

1  NAVAL  SAFETY  CTR 

CODE  471 
D  C  WEIGHTMAN 
NORFOLK  VA  23501 

1  NAVAL  POSTGRADUATE  SCHOOL 
CODE  AA  BP 
ATTN  ROBERT  E  BALL 
MONTEREY  CA  93943-5000 


147 


No  of 

Copies  Organization 

1  NAWC 
526E00D 

1  ADMINSTRATION  CIRCLE 
ATTN  DAVID  SCHRINER 
CHINA  LAKE  CA  93555-6100 

1  DEP  DIR 

HQ  US  AIR  FORCE  T&E 
1650  AIR  FORCE  PENTAGON 
ATTN  JOHN  MANCLARK 
WASHINGTON  DC  20330-1650 

1  DEP  DIR 

HQ  US  AIR  FORCE  T&E 
1650  AIR  FORCE  PENTAGON 
RM  4E995 

LTG  HOWARD  LEAF  USAF  RET 
WASHINGTON  DC  20330-1650 

1  USAF  PHILLIPS  LABORATORY 
ATTN  PL  WS 

3550  ABERDEEN  AVE  SE 
ATTN  WILLIAM  BAKER 
KIRKLAND  AFB  NM  87117-5576 

1  USAF  WRIGHT  LABORATORY 
WLFIVS 

ATTN  MARTIN  LENTZ 
WRIGHT  PAT  AFB  OH  45433-6553 

1  USAF  WRIGHT  LABORATORY 
ARMAMENT  DIRECTORATE 
TECH  MGMT  DIVWLMNP 
WALTER  MAINE 
101  EGLIN  BVLD  SUITE  302 
EGLINAFBFL  32542-6810 

1  USAF  WRIGHT  LABORATORY 

1901  TENTH  ST 
ATTN  RALPH  LAUZZE 
WRIGHT  PAT  AFB  OH  45433-7605 

1  USAF  WRIGHT  LABORATORY 
DIRECTOR  OF  ARMAMENT 
ATTN  WL  MN 
COL  JOHN  PLETCHER 
EGLINAFBFL  32542 

1  USAFASCXRA 
BLDG  16 

2275DSTSUITE10 
HUGH  GRIFFIS 

WRIGHT  PAT  AFB  OH  45433-7227 


No  of 

Copies  Organization 

1  USAFWLFIV 
ST  BLDG  63 
1901  TENTH  ST 
DONALD  VOYLS 

WRIGTH  PAT  AFB  OH  45433-7605 

1  USAF  WL  MNS 

101  WEST  EGLIN  BOULEVARD 
SUITE  302 
LANNYGBURDGE 
EGLIN  AFB  FL  32542-681 0 

1  HQ  USAF  SPECIAL  OPERATIONS  CMD 
AFSOC  LGMW 
100  BARLEY  ST 
ATTN  MSGT  PAUL  PEACOCK 
HURLBURT  FIELD  FL  32536 

1  USAFOPERTEST&EVALCMD 
ATTNAFOCTECN 
MARION  WILLIAMS 
8500  GIBSON  BVLD  SE 
KIRKLAND  AFB  NM  87117-5558 

1  USA  AVIATION  LOGISTICS  SCHOOL 
AVIATION  TRADES  TRAINING 
BLDG  271 5F 

ATTN  LTC  ALPHONSO  MATHEWSON 
FT  EUSITA  VA  23604-5439 

4  ARMY  RESEARCH  OFFICE 
MATH  AND  CS  DIV 
PO  BOX  1221 

ATTN  DR  JAGDISH  CHANDRA 

K  CLARK 

DRWU 

DR  MING  LING 

RES  TRI  PARK  NC  27709-221 1 

1  DEFENSE  NUCLEAR  AGENCY 
WE  MICHAEL  GILTRUDE 
6801  TELEGRAPH  RD 
ALEXANDRIA  VA  2231 0-3398 

1  US  GAO 

DIR  PROG  EVAL  PHYS  SYS 
441  G  STREET  NW 
RM  4062  KWAI-CHUENG  CHAN 
WASHINGTON  DC  20548 

1  LAWRENCE  LIVERMORE 
NATIONAL  LABORATORY 
PO  BOX  808  LI  59 
ATTN  MILTON  FINGER 
LIVERMORE  CA  94551 
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1  LOS  ALAMOS  NATIONAL  LAB 
CONVWEAPON  TECHNOLOGIES 
PO  BOX  1663 
MAIL  STOP  A1 33 
JOSEPH  REPA 
LOS  ALAMOS  NM  87545 

1  LOS  ALAMOS  NATIONAL  LAB 
LANLD-DO  MSB218 
GENE  MCCALL 
LOS  ALAMOS  NM  87545 

1  USAMICOM 

ATTNAMSMI-RD 
PAUL  L  JACOBS 

REDSTN  ARSENAL  AL  35898-5240 

1  CMDRUSAOPTEC 
4501  FORDAVE 
CSTEZTHANKDUBIN 
ALEXANDRIA  VA  22302- 1 458 

1  CMDRUSAOPTEC 
4501  FORDAVE 
CSTEZTLTC  CRABTREE 
PARK  CENTER  IV 
ALEXANDRIA  VA  22302- 1 458 

1  US  SPACE  &  STRATEGIC  DEFENSE 
CMD 

ATTN  HELSTFCSSD  HD 
COL  LARRY  ANDERSON 
WSMR  NM  88002-5148 

1  USATEXCOM 
ATTN  CSTETTD 
BRIAN  BARR  TECHNICAL  DIR 
FT  HOOD  TX  76544-5065 

1  NWSCDL 

JTCG  ME  PROGRAM  OFFICE  G202 
BLDG  221  RM  245 
ATTN  GEORGE  A  WILLIAMS 
DAHLGREN  VA  22448-51 00 

1  US  MILITARY  ACADEMY 
DEPT  OF  MATH  SCI 
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